{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommender Systems 2020/21\n",
    "\n",
    "### Practice - Hyperparameter optimization with Skopt\n",
    "\n",
    "### Hyperparameter optimization is essential to achieve the best recommendation quality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter optimization strategies\n",
    "* Grid-search (use predefined list of values): simple but often unpractically slow and very difficult to fine tune\n",
    "* Random-search (use predefined range and distribution): very parallelizable and effective\n",
    "* Bayesian-search (random search + gaussian process): less parallelizable, good exploration-exploitation tradeoff\n",
    "\n",
    "\n",
    "### In the course repo you will find a BayesianSearch object in the ParameterTuning folder. That is a simple wrapper of another library and its purpose is to provide a very simple way to tune some of the most common parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movielens10M: Verifying data consistency...\n",
      "Movielens10M: Verifying data consistency... Passed!\n",
      "DataReader: current dataset is: <class 'Data_manager.Dataset.Dataset'>\n",
      "\tNumber of items: 10681\n",
      "\tNumber of users: 69878\n",
      "\tNumber of interactions in URM_all: 10000054\n",
      "\tValue range in URM_all: 0.50-5.00\n",
      "\tInteraction density: 1.34E-02\n",
      "\tInteractions per user:\n",
      "\t\t Min: 2.00E+01\n",
      "\t\t Avg: 1.43E+02\n",
      "\t\t Max: 7.36E+03\n",
      "\tInteractions per item:\n",
      "\t\t Min: 0.00E+00\n",
      "\t\t Avg: 9.36E+02\n",
      "\t\t Max: 3.49E+04\n",
      "\tGini Index: 0.57\n",
      "\n",
      "\tICM name: ICM_genres, Value range: 1.00 / 1.00, Num features: 20, feature occurrences: 21564, density 1.01E-01\n",
      "\tICM name: ICM_tags, Value range: 1.00 / 69.00, Num features: 10217, feature occurrences: 108563, density 9.95E-04\n",
      "\tICM name: ICM_all, Value range: 1.00 / 69.00, Num features: 10237, feature occurrences: 130127, density 1.19E-03\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from Notebooks_utils.data_splitter import train_test_holdout\n",
    "from Data_manager.Movielens.Movielens10MReader import Movielens10MReader\n",
    "\n",
    "data_reader = Movielens10MReader()\n",
    "data_loaded = data_reader.load_data()\n",
    "\n",
    "URM_all = data_loaded.get_URM_all()\n",
    "ICM_all = data_loaded.get_ICM_from_name(\"ICM_genres\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How do we perform hyperparameter optimization?\n",
    "* Split the data in three *disjoint* sets: training, validation and testing data\n",
    "* Define a set of hyperparameters with the range and distribution\n",
    "* Explore hyperparameter space and select those with the best recommendation quality on the *validation* data (including the number of epochs for ML algorithms)\n",
    "* Given the best hyperparameters, fit the model again using the union of training and validation data.\n",
    "* Evaluate this last model on the testing data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Split the data and create the evaluator objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Base.Evaluation.Evaluator import EvaluatorHoldout\n",
    "\n",
    "URM_train_validation, URM_test = train_test_holdout(URM_all, train_perc = 0.8)\n",
    "URM_train, URM_validation = train_test_holdout(URM_train_validation, train_perc = 0.8)\n",
    "\n",
    "evaluator_validation = EvaluatorHoldout(URM_validation, cutoff_list=[10])\n",
    "evaluator_test = EvaluatorHoldout(URM_test, cutoff_list=[10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Define hyperparameter set for the desired model, in this case ItemBasedKNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skopt.space import Real, Integer, Categorical\n",
    "\n",
    "hyperparameters_range_dictionary = {}\n",
    "hyperparameters_range_dictionary[\"topK\"] = Integer(5, 1000)\n",
    "hyperparameters_range_dictionary[\"shrink\"] = Integer(0, 1000)\n",
    "hyperparameters_range_dictionary[\"similarity\"] = Categorical([\"cosine\"])\n",
    "hyperparameters_range_dictionary[\"normalize\"] = Categorical([True, False])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Create SearchBayesianSkopt object, providing the desired recommender class and evaluator objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from KNN.ItemKNNCFRecommender import ItemKNNCFRecommender\n",
    "from ParameterTuning.SearchBayesianSkopt import SearchBayesianSkopt\n",
    "\n",
    "recommender_class = ItemKNNCFRecommender\n",
    "\n",
    "parameterSearch = SearchBayesianSkopt(recommender_class,\n",
    "                                 evaluator_validation=evaluator_validation,\n",
    "                                 evaluator_test=evaluator_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Provide the data needed to create an instance of the model, one trained only on URM_train, the other on URM_train_validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ParameterTuning.SearchAbstractClass import SearchInputRecommenderArgs\n",
    "  \n",
    "recommender_input_args = SearchInputRecommenderArgs(\n",
    "    CONSTRUCTOR_POSITIONAL_ARGS = [URM_train],     # For a CBF model simply put [URM_train, ICM_train]\n",
    "    CONSTRUCTOR_KEYWORD_ARGS = {},\n",
    "    FIT_POSITIONAL_ARGS = [],\n",
    "    FIT_KEYWORD_ARGS = {}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "recommender_input_args_last_test = SearchInputRecommenderArgs(\n",
    "    CONSTRUCTOR_POSITIONAL_ARGS = [URM_train_validation],     # For a CBF model simply put [URM_train_validation, ICM_train]\n",
    "    CONSTRUCTOR_KEYWORD_ARGS = {},\n",
    "    FIT_POSITIONAL_ARGS = [],\n",
    "    FIT_KEYWORD_ARGS = {}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Create a result folder and select the number of cases (50 with 30% random is a good number)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "output_folder_path = \"result_experiments/\"\n",
    "\n",
    "# If directory does not exist, create\n",
    "if not os.path.exists(output_folder_path):\n",
    "    os.makedirs(output_folder_path)\n",
    "    \n",
    "n_cases = 10\n",
    "n_random_starts = int(n_cases*0.3)\n",
    "metric_to_optimize = \"MAP\"   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Run!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "ItemKNNCFRecommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 789, 'shrink': 259, 'similarity': 'cosine', 'normalize': True}\n",
      "Similarity column 10681 ( 100 % ), 987.88 column/sec, elapsed time 0.18 min\n",
      "EvaluatorHoldout: Processed 32000 ( 45.95% ) in 30.93 sec. Users per second: 1034\n",
      "EvaluatorHoldout: Processed 63000 ( 90.47% ) in 1.03 min. Users per second: 1022\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 1.13 min. Users per second: 1023\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'topK': 789, 'shrink': 259, 'similarity': 'cosine', 'normalize': True} - results: ROC_AUC: 0.4390522, PRECISION: 0.1906800, PRECISION_RECALL_MIN_DEN: 0.2388892, RECALL: 0.1559166, MAP: 0.1239337, MRR: 0.4154496, NDCG: 0.1649812, F1: 0.1715549, HIT_RATE: 1.9067997, ARHR: 0.6396330, NOVELTY: 0.0086132, AVERAGE_POPULARITY: 0.5370173, DIVERSITY_MEAN_INTER_LIST: 0.8917573, DIVERSITY_HERFINDAHL: 0.9891745, COVERAGE_ITEM: 0.1761071, COVERAGE_ITEM_CORRECT: 0.0864151, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.7599960, DIVERSITY_GINI: 0.0114700, SHANNON_ENTROPY: 7.2527011, \n",
      "\n",
      "EvaluatorHoldout: Processed 31000 ( 44.42% ) in 30.48 sec. Users per second: 1017\n",
      "EvaluatorHoldout: Processed 62000 ( 88.84% ) in 1.02 min. Users per second: 1010\n",
      "EvaluatorHoldout: Processed 69786 ( 100.00% ) in 1.15 min. Users per second: 1008\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'topK': 789, 'shrink': 259, 'similarity': 'cosine', 'normalize': True} - results:\n",
      "CUTOFF: 10 - ROC_AUC: 0.4698926, PRECISION: 0.2378873, PRECISION_RECALL_MIN_DEN: 0.2736271, RECALL: 0.1554251, MAP: 0.1557973, MRR: 0.4801354, NDCG: 0.1783134, F1: 0.1880117, HIT_RATE: 2.3788726, ARHR: 0.7958696, NOVELTY: 0.0086133, AVERAGE_POPULARITY: 0.5370544, DIVERSITY_MEAN_INTER_LIST: 0.8917500, DIVERSITY_HERFINDAHL: 0.9891737, COVERAGE_ITEM: 0.1770433, COVERAGE_ITEM_CORRECT: 0.0948413, COVERAGE_USER: 0.9986834, COVERAGE_USER_CORRECT: 0.8084948, DIVERSITY_GINI: 0.0114830, SHANNON_ENTROPY: 7.2533046, \n",
      "\n",
      "\n",
      "DataIO: Json dumps supports only 'str' as dictionary keys. Transforming keys to string, note that this will alter the mapper content.\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 149.0399\n",
      "Function value obtained: -0.1239\n",
      "Current minimum: -0.1239\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "ItemKNNCFRecommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 670, 'shrink': 121, 'similarity': 'cosine', 'normalize': True}\n",
      "Similarity column 10681 ( 100 % ), 1007.53 column/sec, elapsed time 0.18 min\n",
      "EvaluatorHoldout: Processed 34000 ( 48.83% ) in 30.54 sec. Users per second: 1113\n",
      "EvaluatorHoldout: Processed 69000 ( 99.09% ) in 1.02 min. Users per second: 1128\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 1.03 min. Users per second: 1124\n",
      "SearchBayesianSkopt: New best config found. Config 1: {'topK': 670, 'shrink': 121, 'similarity': 'cosine', 'normalize': True} - results: ROC_AUC: 0.4383538, PRECISION: 0.1919193, PRECISION_RECALL_MIN_DEN: 0.2402578, RECALL: 0.1566718, MAP: 0.1245181, MRR: 0.4159570, NDCG: 0.1653922, F1: 0.1725135, HIT_RATE: 1.9191929, ARHR: 0.6417900, NOVELTY: 0.0086356, AVERAGE_POPULARITY: 0.5317092, DIVERSITY_MEAN_INTER_LIST: 0.8977719, DIVERSITY_HERFINDAHL: 0.9897759, COVERAGE_ITEM: 0.1919296, COVERAGE_ITEM_CORRECT: 0.0944668, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.7611408, DIVERSITY_GINI: 0.0123937, SHANNON_ENTROPY: 7.3503141, \n",
      "\n",
      "EvaluatorHoldout: Processed 31000 ( 44.42% ) in 30.26 sec. Users per second: 1025\n",
      "EvaluatorHoldout: Processed 64000 ( 91.71% ) in 1.01 min. Users per second: 1061\n",
      "EvaluatorHoldout: Processed 69786 ( 100.00% ) in 1.09 min. Users per second: 1063\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'topK': 670, 'shrink': 121, 'similarity': 'cosine', 'normalize': True} - results:\n",
      "CUTOFF: 10 - ROC_AUC: 0.4700438, PRECISION: 0.2390494, PRECISION_RECALL_MIN_DEN: 0.2749585, RECALL: 0.1561602, MAP: 0.1566572, MRR: 0.4806349, NDCG: 0.1787179, F1: 0.1889124, HIT_RATE: 2.3904938, ARHR: 0.7987476, NOVELTY: 0.0086357, AVERAGE_POPULARITY: 0.5317612, DIVERSITY_MEAN_INTER_LIST: 0.8977607, DIVERSITY_HERFINDAHL: 0.9897748, COVERAGE_ITEM: 0.1929595, COVERAGE_ITEM_CORRECT: 0.1023312, COVERAGE_USER: 0.9986834, COVERAGE_USER_CORRECT: 0.8085377, DIVERSITY_GINI: 0.0124066, SHANNON_ENTROPY: 7.3507962, \n",
      "\n",
      "\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 139.0304\n",
      "Function value obtained: -0.1245\n",
      "Current minimum: -0.1245\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "ItemKNNCFRecommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 80, 'shrink': 487, 'similarity': 'cosine', 'normalize': False}\n",
      "Similarity column 10681 ( 100 % ), 934.19 column/sec, elapsed time 0.19 min\n",
      "EvaluatorHoldout: Processed 34000 ( 48.83% ) in 30.34 sec. Users per second: 1121\n",
      "EvaluatorHoldout: Processed 69000 ( 99.09% ) in 1.02 min. Users per second: 1132\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 1.03 min. Users per second: 1132\n",
      "SearchBayesianSkopt: Config 2 is suboptimal. Config: {'topK': 80, 'shrink': 487, 'similarity': 'cosine', 'normalize': False} - results: ROC_AUC: 0.3917837, PRECISION: 0.1536311, PRECISION_RECALL_MIN_DEN: 0.1902464, RECALL: 0.1214750, MAP: 0.0939908, MRR: 0.3535743, NDCG: 0.1376485, F1: 0.1356737, HIT_RATE: 1.5363108, ARHR: 0.5165515, NOVELTY: 0.0081040, AVERAGE_POPULARITY: 0.7410513, DIVERSITY_MEAN_INTER_LIST: 0.6830844, DIVERSITY_HERFINDAHL: 0.9683075, COVERAGE_ITEM: 0.1482071, COVERAGE_ITEM_CORRECT: 0.0282745, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.6861816, DIVERSITY_GINI: 0.0040239, SHANNON_ENTROPY: 5.5592929, \n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 73.8752\n",
      "Function value obtained: -0.0940\n",
      "Current minimum: -0.1245\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "ItemKNNCFRecommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 5, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}\n",
      "Similarity column 10681 ( 100 % ), 927.20 column/sec, elapsed time 0.19 min\n",
      "EvaluatorHoldout: Processed 60000 ( 86.16% ) in 30.14 sec. Users per second: 1991\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.06 sec. Users per second: 1986\n",
      "SearchBayesianSkopt: Config 3 is suboptimal. Config: {'topK': 5, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True} - results: ROC_AUC: 0.4138974, PRECISION: 0.1636806, PRECISION_RECALL_MIN_DEN: 0.2051340, RECALL: 0.1333298, MAP: 0.1005795, MRR: 0.3727609, NDCG: 0.1404112, F1: 0.1469545, HIT_RATE: 1.6368062, ARHR: 0.5474097, NOVELTY: 0.0090174, AVERAGE_POPULARITY: 0.4408844, DIVERSITY_MEAN_INTER_LIST: 0.9540680, DIVERSITY_HERFINDAHL: 0.9954054, COVERAGE_ITEM: 0.2894860, COVERAGE_ITEM_CORRECT: 0.1468964, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.7251925, DIVERSITY_GINI: 0.0313373, SHANNON_ENTROPY: 8.6631770, \n",
      "\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 47.5260\n",
      "Function value obtained: -0.1006\n",
      "Current minimum: -0.1245\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "ItemKNNCFRecommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 1000, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}\n",
      "Similarity column 10681 ( 100 % ), 924.24 column/sec, elapsed time 0.19 min\n",
      "EvaluatorHoldout: Processed 22000 ( 31.59% ) in 31.30 sec. Users per second: 703\n",
      "EvaluatorHoldout: Processed 45000 ( 64.62% ) in 1.04 min. Users per second: 723\n",
      "EvaluatorHoldout: Processed 68000 ( 97.65% ) in 1.55 min. Users per second: 732\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 1.58 min. Users per second: 732\n",
      "SearchBayesianSkopt: Config 4 is suboptimal. Config: {'topK': 1000, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True} - results: ROC_AUC: 0.4363987, PRECISION: 0.1886709, PRECISION_RECALL_MIN_DEN: 0.2365778, RECALL: 0.1544887, MAP: 0.1225033, MRR: 0.4120079, NDCG: 0.1635640, F1: 0.1698774, HIT_RATE: 1.8867093, ARHR: 0.6329142, NOVELTY: 0.0085689, AVERAGE_POPULARITY: 0.5482694, DIVERSITY_MEAN_INTER_LIST: 0.8791718, DIVERSITY_HERFINDAHL: 0.9879159, COVERAGE_ITEM: 0.1374403, COVERAGE_ITEM_CORRECT: 0.0736822, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.7562466, DIVERSITY_GINI: 0.0098889, SHANNON_ENTROPY: 7.0675385, \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 108.1322\n",
      "Function value obtained: -0.1225\n",
      "Current minimum: -0.1245\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "ItemKNNCFRecommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': True}\n",
      "Similarity column 10681 ( 100 % ), 947.15 column/sec, elapsed time 0.19 min\n",
      "EvaluatorHoldout: Processed 30000 ( 43.08% ) in 30.12 sec. Users per second: 996\n",
      "EvaluatorHoldout: Processed 60000 ( 86.16% ) in 1.00 min. Users per second: 995\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 1.17 min. Users per second: 991\n",
      "SearchBayesianSkopt: Config 5 is suboptimal. Config: {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': True} - results: ROC_AUC: 0.4372607, PRECISION: 0.1892051, PRECISION_RECALL_MIN_DEN: 0.2373137, RECALL: 0.1550236, MAP: 0.1228231, MRR: 0.4126555, NDCG: 0.1637222, F1: 0.1704172, HIT_RATE: 1.8920514, ARHR: 0.6343127, NOVELTY: 0.0085951, AVERAGE_POPULARITY: 0.5419792, DIVERSITY_MEAN_INTER_LIST: 0.8832303, DIVERSITY_HERFINDAHL: 0.9883218, COVERAGE_ITEM: 0.1749836, COVERAGE_ITEM_CORRECT: 0.0832319, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.7578351, DIVERSITY_GINI: 0.0106748, SHANNON_ENTROPY: 7.1452778, \n",
      "\n",
      "Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "Time taken: 82.9723\n",
      "Function value obtained: -0.1228\n",
      "Current minimum: -0.1245\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n",
      "ItemKNNCFRecommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': False}\n",
      "Similarity column 10681 ( 100 % ), 965.13 column/sec, elapsed time 0.18 min\n",
      "EvaluatorHoldout: Processed 18000 ( 25.85% ) in 30.88 sec. Users per second: 583\n",
      "EvaluatorHoldout: Processed 36000 ( 51.70% ) in 1.03 min. Users per second: 584\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 1.54 min. Users per second: 584\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 2.00 min. Users per second: 581\n",
      "SearchBayesianSkopt: Config 6 is suboptimal. Config: {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': False} - results: ROC_AUC: 0.3706175, PRECISION: 0.1400933, PRECISION_RECALL_MIN_DEN: 0.1711721, RECALL: 0.1059862, MAP: 0.0827484, MRR: 0.3245020, NDCG: 0.1230361, F1: 0.1206761, HIT_RATE: 1.4009334, ARHR: 0.4683799, NOVELTY: 0.0079680, AVERAGE_POPULARITY: 0.7970184, DIVERSITY_MEAN_INTER_LIST: 0.5372315, DIVERSITY_HERFINDAHL: 0.9537224, COVERAGE_ITEM: 0.0185376, COVERAGE_ITEM_CORRECT: 0.0071154, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.6504622, DIVERSITY_GINI: 0.0019535, SHANNON_ENTROPY: 4.7096959, \n",
      "\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 132.3829\n",
      "Function value obtained: -0.0827\n",
      "Current minimum: -0.1245\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n",
      "ItemKNNCFRecommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 710, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}\n",
      "Similarity column 10681 ( 100 % ), 875.71 column/sec, elapsed time 0.20 min\n",
      "EvaluatorHoldout: Processed 26000 ( 37.34% ) in 30.22 sec. Users per second: 860\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 1.02 min. Users per second: 868\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 1.33 min. Users per second: 870\n",
      "SearchBayesianSkopt: Config 7 is suboptimal. Config: {'topK': 710, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True} - results: ROC_AUC: 0.4386454, PRECISION: 0.1908925, PRECISION_RECALL_MIN_DEN: 0.2390361, RECALL: 0.1559033, MAP: 0.1240412, MRR: 0.4152712, NDCG: 0.1650181, F1: 0.1716328, HIT_RATE: 1.9089251, ARHR: 0.6400954, NOVELTY: 0.0086066, AVERAGE_POPULARITY: 0.5387098, DIVERSITY_MEAN_INTER_LIST: 0.8927220, DIVERSITY_HERFINDAHL: 0.9892709, COVERAGE_ITEM: 0.1570078, COVERAGE_ITEM_CORRECT: 0.0816403, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.7591946, DIVERSITY_GINI: 0.0113365, SHANNON_ENTROPY: 7.2526984, \n",
      "\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 93.4106\n",
      "Function value obtained: -0.1240\n",
      "Current minimum: -0.1245\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "ItemKNNCFRecommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 734, 'shrink': 0, 'similarity': 'cosine', 'normalize': True}\n",
      "Similarity column 10681 ( 100 % ), 972.88 column/sec, elapsed time 0.18 min\n",
      "EvaluatorHoldout: Processed 34000 ( 48.83% ) in 30.54 sec. Users per second: 1113\n",
      "EvaluatorHoldout: Processed 69000 ( 99.09% ) in 1.02 min. Users per second: 1123\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 1.03 min. Users per second: 1123\n",
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'topK': 734, 'shrink': 0, 'similarity': 'cosine', 'normalize': True} - results: ROC_AUC: 0.4386956, PRECISION: 0.1914325, PRECISION_RECALL_MIN_DEN: 0.2397177, RECALL: 0.1564179, MAP: 0.1243698, MRR: 0.4155637, NDCG: 0.1652361, F1: 0.1721629, HIT_RATE: 1.9143247, ARHR: 0.6410582, NOVELTY: 0.0086298, AVERAGE_POPULARITY: 0.5333855, DIVERSITY_MEAN_INTER_LIST: 0.8952528, DIVERSITY_HERFINDAHL: 0.9895240, COVERAGE_ITEM: 0.1954873, COVERAGE_ITEM_CORRECT: 0.0919390, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.7601105, DIVERSITY_GINI: 0.0120941, SHANNON_ENTROPY: 7.3131478, \n",
      "\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 74.2137\n",
      "Function value obtained: -0.1244\n",
      "Current minimum: -0.1245\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n",
      "ItemKNNCFRecommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 769, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}\n",
      "Similarity column 10681 ( 100 % ), 920.46 column/sec, elapsed time 0.19 min\n",
      "EvaluatorHoldout: Processed 26000 ( 37.34% ) in 30.55 sec. Users per second: 851\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 1.02 min. Users per second: 833\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 1.39 min. Users per second: 834\n",
      "SearchBayesianSkopt: Config 9 is suboptimal. Config: {'topK': 769, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True} - results: ROC_AUC: 0.4387259, PRECISION: 0.1903755, PRECISION_RECALL_MIN_DEN: 0.2383728, RECALL: 0.1554546, MAP: 0.1238408, MRR: 0.4150750, NDCG: 0.1649830, F1: 0.1711519, HIT_RATE: 1.9037553, ARHR: 0.6392349, NOVELTY: 0.0085979, AVERAGE_POPULARITY: 0.5408102, DIVERSITY_MEAN_INTER_LIST: 0.8899969, DIVERSITY_HERFINDAHL: 0.9889984, COVERAGE_ITEM: 0.1519521, COVERAGE_ITEM_CORRECT: 0.0791124, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.7581642, DIVERSITY_GINI: 0.0109878, SHANNON_ENTROPY: 7.2116011, \n",
      "\n",
      "Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "Time taken: 96.4847\n",
      "Function value obtained: -0.1238\n",
      "Current minimum: -0.1245\n",
      "SearchBayesianSkopt: Search complete. Best config is 1: {'topK': 670, 'shrink': 121, 'similarity': 'cosine', 'normalize': True}\n",
      "\n",
      "ItemKNNCFRecommender: URM Detected 37 (0.35 %) cold items.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'topK': 670, 'shrink': 121, 'similarity': 'cosine', 'normalize': True}\n",
      "Similarity column 10681 ( 100 % ), 692.05 column/sec, elapsed time 0.26 min\n",
      "EvaluatorHoldout: Processed 30000 ( 42.99% ) in 30.55 sec. Users per second: 982\n",
      "EvaluatorHoldout: Processed 59000 ( 84.54% ) in 1.02 min. Users per second: 964\n",
      "EvaluatorHoldout: Processed 69786 ( 100.00% ) in 1.21 min. Users per second: 965\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'topK': 670, 'shrink': 121, 'similarity': 'cosine', 'normalize': True} - results:\n",
      "CUTOFF: 10 - ROC_AUC: 0.5334466, PRECISION: 0.3053807, PRECISION_RECALL_MIN_DEN: 0.3461029, RECALL: 0.1848276, MAP: 0.2432645, MRR: 0.5859146, NDCG: 0.2223354, F1: 0.2302808, HIT_RATE: 3.0538074, ARHR: 1.0804503, NOVELTY: 0.0086681, AVERAGE_POPULARITY: 0.5181971, DIVERSITY_MEAN_INTER_LIST: 0.9076256, DIVERSITY_HERFINDAHL: 0.9907613, COVERAGE_ITEM: 0.1829417, COVERAGE_ITEM_CORRECT: 0.1132853, COVERAGE_USER: 0.9986834, COVERAGE_USER_CORRECT: 0.8272131, DIVERSITY_GINI: 0.0131675, SHANNON_ENTROPY: 7.4681829, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCFRecommender\n",
      "\n",
      "ItemKNNCFRecommender: Saving model in file 'result_experiments/ItemKNNCFRecommender_best_model_last'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ItemKNNCFRecommender: Saving complete\n"
     ]
    }
   ],
   "source": [
    "parameterSearch.search(recommender_input_args,\n",
    "                       recommender_input_args_last_test = recommender_input_args_last_test,\n",
    "                       parameter_search_space = hyperparameters_range_dictionary,\n",
    "                       n_cases = n_cases,\n",
    "                       n_random_starts = n_random_starts,\n",
    "                       save_model = \"last\",\n",
    "                       output_folder_path = output_folder_path,\n",
    "                       output_file_name_root = recommender_class.RECOMMENDER_NAME,\n",
    "                       metric_to_optimize = metric_to_optimize,\n",
    "                      )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'algorithm_name_search': 'SearchBayesianSkopt',\n",
       " 'algorithm_name_recommender': 'ItemKNNCFRecommender',\n",
       " 'exception_list': [None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None],\n",
       " 'hyperparameters_list': [{'topK': 789,\n",
       "   'shrink': 259,\n",
       "   'similarity': 'cosine',\n",
       "   'normalize': True},\n",
       "  {'topK': 670, 'shrink': 121, 'similarity': 'cosine', 'normalize': True},\n",
       "  {'topK': 80, 'shrink': 487, 'similarity': 'cosine', 'normalize': False},\n",
       "  {'topK': 5, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True},\n",
       "  {'topK': 1000, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True},\n",
       "  {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': True},\n",
       "  {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': False},\n",
       "  {'topK': 710, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True},\n",
       "  {'topK': 734, 'shrink': 0, 'similarity': 'cosine', 'normalize': True},\n",
       "  {'topK': 769, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}],\n",
       " 'hyperparameters_best': {'topK': 670,\n",
       "  'shrink': 121,\n",
       "  'similarity': 'cosine',\n",
       "  'normalize': True},\n",
       " 'hyperparameters_best_index': 1,\n",
       " 'result_on_validation_list': [{'ROC_AUC': 0.43905222811462563,\n",
       "   'PRECISION': 0.19067997415097757,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.23888923080783037,\n",
       "   'RECALL': 0.15591662380462407,\n",
       "   'MAP': 0.12393373341827027,\n",
       "   'MRR': 0.4154496347736068,\n",
       "   'NDCG': 0.16498123127539516,\n",
       "   'F1': 0.17155493142250516,\n",
       "   'HIT_RATE': 1.9067997415092985,\n",
       "   'ARHR': 0.6396330184259441,\n",
       "   'NOVELTY': 0.008613206413749457,\n",
       "   'AVERAGE_POPULARITY': 0.5370172593120054,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.8917573414899574,\n",
       "   'DIVERSITY_HERFINDAHL': 0.9891744535324358,\n",
       "   'COVERAGE_ITEM': 0.17610710607621008,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.0864151296695066,\n",
       "   'COVERAGE_USER': 0.9965225106614385,\n",
       "   'COVERAGE_USER_CORRECT': 0.7599959930164,\n",
       "   'DIVERSITY_GINI': 0.01147003646894625,\n",
       "   'SHANNON_ENTROPY': 7.252701083064463},\n",
       "  {'ROC_AUC': 0.43835377096675077,\n",
       "   'PRECISION': 0.1919192934587982,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.24025778976775858,\n",
       "   'RECALL': 0.15667181356169146,\n",
       "   'MAP': 0.12451814778750853,\n",
       "   'MRR': 0.41595699685776216,\n",
       "   'NDCG': 0.1653921751381285,\n",
       "   'F1': 0.1725135447124359,\n",
       "   'HIT_RATE': 1.9191929345874918,\n",
       "   'ARHR': 0.6417900424092761,\n",
       "   'NOVELTY': 0.008635632211593499,\n",
       "   'AVERAGE_POPULARITY': 0.5317092123580478,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.8977718753710007,\n",
       "   'DIVERSITY_HERFINDAHL': 0.9897758982833119,\n",
       "   'COVERAGE_ITEM': 0.1919295946072465,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.09446681022376183,\n",
       "   'COVERAGE_USER': 0.9965225106614385,\n",
       "   'COVERAGE_USER_CORRECT': 0.7611408454735396,\n",
       "   'DIVERSITY_GINI': 0.01239368700090733,\n",
       "   'SHANNON_ENTROPY': 7.350314089525069},\n",
       "  {'ROC_AUC': 0.391783702662753,\n",
       "   'PRECISION': 0.15363107632660705,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.19024635827863032,\n",
       "   'RECALL': 0.12147499414003392,\n",
       "   'MAP': 0.09399082580664395,\n",
       "   'MRR': 0.3535743006903426,\n",
       "   'NDCG': 0.13764854090238687,\n",
       "   'F1': 0.13567373533303886,\n",
       "   'HIT_RATE': 1.5363107632655992,\n",
       "   'ARHR': 0.5165514684847228,\n",
       "   'NOVELTY': 0.008103955679769159,\n",
       "   'AVERAGE_POPULARITY': 0.7410513348622169,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.6830843642197776,\n",
       "   'DIVERSITY_HERFINDAHL': 0.9683074554722194,\n",
       "   'COVERAGE_ITEM': 0.14820709671379084,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.028274506132384608,\n",
       "   'COVERAGE_USER': 0.9965225106614385,\n",
       "   'COVERAGE_USER_CORRECT': 0.6861816308423252,\n",
       "   'DIVERSITY_GINI': 0.004023875960492474,\n",
       "   'SHANNON_ENTROPY': 5.559292882117824},\n",
       "  {'ROC_AUC': 0.4138973821547985,\n",
       "   'PRECISION': 0.16368062037773318,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.20513399802374116,\n",
       "   'RECALL': 0.13332984386836022,\n",
       "   'MAP': 0.10057954045301973,\n",
       "   'MRR': 0.3727609439697555,\n",
       "   'NDCG': 0.14041122065228057,\n",
       "   'F1': 0.14695449612951844,\n",
       "   'HIT_RATE': 1.6368062037768363,\n",
       "   'ARHR': 0.5474097419537444,\n",
       "   'NOVELTY': 0.00901737084531927,\n",
       "   'AVERAGE_POPULARITY': 0.4408843523279395,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.9540679853197247,\n",
       "   'DIVERSITY_HERFINDAHL': 0.9954054284336235,\n",
       "   'COVERAGE_ITEM': 0.28948600318322254,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.1468963580189121,\n",
       "   'COVERAGE_USER': 0.9965225106614385,\n",
       "   'COVERAGE_USER_CORRECT': 0.7251924783193566,\n",
       "   'DIVERSITY_GINI': 0.0313373030074825,\n",
       "   'SHANNON_ENTROPY': 8.66317698043621},\n",
       "  {'ROC_AUC': 0.4363986803069479,\n",
       "   'PRECISION': 0.18867092697642565,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.2365777620495327,\n",
       "   'RECALL': 0.1544886991221431,\n",
       "   'MAP': 0.12250331758362838,\n",
       "   'MRR': 0.41200785615699087,\n",
       "   'NDCG': 0.16356402534456346,\n",
       "   'F1': 0.16987736233507,\n",
       "   'HIT_RATE': 1.8867092697637682,\n",
       "   'ARHR': 0.6329141635351939,\n",
       "   'NOVELTY': 0.008568946156455025,\n",
       "   'AVERAGE_POPULARITY': 0.5482694248670622,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.8791718114344513,\n",
       "   'DIVERSITY_HERFINDAHL': 0.9879159186004546,\n",
       "   'COVERAGE_ITEM': 0.13744031457728678,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.07368223949068439,\n",
       "   'COVERAGE_USER': 0.9965225106614385,\n",
       "   'COVERAGE_USER_CORRECT': 0.7562466012192679,\n",
       "   'DIVERSITY_GINI': 0.009888947110704711,\n",
       "   'SHANNON_ENTROPY': 7.067538479380985},\n",
       "  {'ROC_AUC': 0.4372607160237993,\n",
       "   'PRECISION': 0.18920514109288974,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.23731373682047552,\n",
       "   'RECALL': 0.15502355665446238,\n",
       "   'MAP': 0.12282309516835785,\n",
       "   'MRR': 0.41265549047698663,\n",
       "   'NDCG': 0.16372223506906594,\n",
       "   'F1': 0.17041724935471197,\n",
       "   'HIT_RATE': 1.8920514109284123,\n",
       "   'ARHR': 0.6343126517976823,\n",
       "   'NOVELTY': 0.008595136261261979,\n",
       "   'AVERAGE_POPULARITY': 0.5419791610881217,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.8832303176770193,\n",
       "   'DIVERSITY_HERFINDAHL': 0.9883217633964552,\n",
       "   'COVERAGE_ITEM': 0.17498361576631402,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.08323190712480105,\n",
       "   'COVERAGE_USER': 0.9965225106614385,\n",
       "   'COVERAGE_USER_CORRECT': 0.757835084003549,\n",
       "   'DIVERSITY_GINI': 0.010674756638538391,\n",
       "   'SHANNON_ENTROPY': 7.14527780529064},\n",
       "  {'ROC_AUC': 0.37061752437027146,\n",
       "   'PRECISION': 0.14009334386447708,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.17117213224059163,\n",
       "   'RECALL': 0.10598620807385817,\n",
       "   'MAP': 0.08274842812275833,\n",
       "   'MRR': 0.3245019609050042,\n",
       "   'NDCG': 0.12303609273492452,\n",
       "   'F1': 0.12067611612283625,\n",
       "   'HIT_RATE': 1.40093343864436,\n",
       "   'ARHR': 0.4683798855939172,\n",
       "   'NOVELTY': 0.007968048748802038,\n",
       "   'AVERAGE_POPULARITY': 0.7970184463293107,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.537231504144992,\n",
       "   'DIVERSITY_HERFINDAHL': 0.9537223789181193,\n",
       "   'COVERAGE_ITEM': 0.018537590113285273,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.007115438629341822,\n",
       "   'COVERAGE_USER': 0.9965225106614385,\n",
       "   'COVERAGE_USER_CORRECT': 0.65046223417957,\n",
       "   'DIVERSITY_GINI': 0.0019534990182781924,\n",
       "   'SHANNON_ENTROPY': 4.7096959058369725},\n",
       "  {'ROC_AUC': 0.4386453628386553,\n",
       "   'PRECISION': 0.19089251095000145,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.23903606218823673,\n",
       "   'RECALL': 0.15590330491175455,\n",
       "   'MAP': 0.1240411694754074,\n",
       "   'MRR': 0.41527117019469856,\n",
       "   'NDCG': 0.16501811116185,\n",
       "   'F1': 0.17163282818770872,\n",
       "   'HIT_RATE': 1.9089251094995332,\n",
       "   'ARHR': 0.6400954295697686,\n",
       "   'NOVELTY': 0.008606564949761123,\n",
       "   'AVERAGE_POPULARITY': 0.5387098206658315,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.8927220243367511,\n",
       "   'DIVERSITY_HERFINDAHL': 0.9892709204317733,\n",
       "   'COVERAGE_ITEM': 0.1570077708079768,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.08164029585244827,\n",
       "   'COVERAGE_USER': 0.9965225106614385,\n",
       "   'COVERAGE_USER_CORRECT': 0.7591945962964023,\n",
       "   'DIVERSITY_GINI': 0.011336528379644325,\n",
       "   'SHANNON_ENTROPY': 7.252698391587996},\n",
       "  {'ROC_AUC': 0.43869555083707185,\n",
       "   'PRECISION': 0.1914324693042779,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.23971770604321302,\n",
       "   'RECALL': 0.15641788006677337,\n",
       "   'MAP': 0.1243697524288171,\n",
       "   'MRR': 0.41556366473254114,\n",
       "   'NDCG': 0.16523606795903292,\n",
       "   'F1': 0.1721628917645972,\n",
       "   'HIT_RATE': 1.914324693042292,\n",
       "   'ARHR': 0.6410581649666938,\n",
       "   'NOVELTY': 0.008629821476984453,\n",
       "   'AVERAGE_POPULARITY': 0.5333854841177535,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.8952527643541246,\n",
       "   'DIVERSITY_HERFINDAHL': 0.9895239907992175,\n",
       "   'COVERAGE_ITEM': 0.19548731392191743,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.09193895702649564,\n",
       "   'COVERAGE_USER': 0.9965225106614385,\n",
       "   'COVERAGE_USER_CORRECT': 0.760110478262114,\n",
       "   'DIVERSITY_GINI': 0.012094091551035222,\n",
       "   'SHANNON_ENTROPY': 7.313147760331327},\n",
       "  {'ROC_AUC': 0.43872591152733864,\n",
       "   'PRECISION': 0.19037552954697035,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.23837281357101409,\n",
       "   'RECALL': 0.15545456941235225,\n",
       "   'MAP': 0.12384084307807192,\n",
       "   'MRR': 0.41507495432532804,\n",
       "   'NDCG': 0.16498299876109762,\n",
       "   'F1': 0.17115193883603422,\n",
       "   'HIT_RATE': 1.9037552954692325,\n",
       "   'ARHR': 0.6392348652439948,\n",
       "   'NOVELTY': 0.00859792063718599,\n",
       "   'AVERAGE_POPULARITY': 0.5408101917820353,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.8899969462958991,\n",
       "   'DIVERSITY_HERFINDAHL': 0.9889984165410621,\n",
       "   'COVERAGE_ITEM': 0.15195206441344444,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.0791124426551821,\n",
       "   'COVERAGE_USER': 0.9965225106614385,\n",
       "   'COVERAGE_USER_CORRECT': 0.7581642290849767,\n",
       "   'DIVERSITY_GINI': 0.010987847982096498,\n",
       "   'SHANNON_ENTROPY': 7.211601059371816}],\n",
       " 'result_on_validation_best': {'ROC_AUC': 0.43835377096675077,\n",
       "  'PRECISION': 0.1919192934587982,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.24025778976775858,\n",
       "  'RECALL': 0.15667181356169146,\n",
       "  'MAP': 0.12451814778750853,\n",
       "  'MRR': 0.41595699685776216,\n",
       "  'NDCG': 0.1653921751381285,\n",
       "  'F1': 0.1725135447124359,\n",
       "  'HIT_RATE': 1.9191929345874918,\n",
       "  'ARHR': 0.6417900424092761,\n",
       "  'NOVELTY': 0.008635632211593499,\n",
       "  'AVERAGE_POPULARITY': 0.5317092123580478,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.8977718753710007,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9897758982833119,\n",
       "  'COVERAGE_ITEM': 0.1919295946072465,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.09446681022376183,\n",
       "  'COVERAGE_USER': 0.9965225106614385,\n",
       "  'COVERAGE_USER_CORRECT': 0.7611408454735396,\n",
       "  'DIVERSITY_GINI': 0.01239368700090733,\n",
       "  'SHANNON_ENTROPY': 7.350314089525069},\n",
       " 'result_on_test_list': [{'10': {'ROC_AUC': 0.4698925774897292,\n",
       "    'PRECISION': 0.23788725532345958,\n",
       "    'PRECISION_RECALL_MIN_DEN': 0.2736270612334714,\n",
       "    'RECALL': 0.15542512371325773,\n",
       "    'MAP': 0.15579730492998972,\n",
       "    'MRR': 0.4801354389996703,\n",
       "    'NDCG': 0.17831341361173533,\n",
       "    'F1': 0.18801165719223087,\n",
       "    'HIT_RATE': 2.378872553234173,\n",
       "    'ARHR': 0.7958695892976344,\n",
       "    'NOVELTY': 0.00861328616767647,\n",
       "    'AVERAGE_POPULARITY': 0.537054409959281,\n",
       "    'DIVERSITY_MEAN_INTER_LIST': 0.8917500269983711,\n",
       "    'DIVERSITY_HERFINDAHL': 0.9891737248647026,\n",
       "    'COVERAGE_ITEM': 0.17704334800112348,\n",
       "    'COVERAGE_ITEM_CORRECT': 0.09484130699372718,\n",
       "    'COVERAGE_USER': 0.9986834196742895,\n",
       "    'COVERAGE_USER_CORRECT': 0.8084948052319757,\n",
       "    'DIVERSITY_GINI': 0.011483033162183416,\n",
       "    'SHANNON_ENTROPY': 7.2533046050566305}},\n",
       "  {'10': {'ROC_AUC': 0.4700438449245519,\n",
       "    'PRECISION': 0.23904937953175287,\n",
       "    'PRECISION_RECALL_MIN_DEN': 0.27495851262295795,\n",
       "    'RECALL': 0.15616019154465238,\n",
       "    'MAP': 0.15665723144248708,\n",
       "    'MRR': 0.48063485694814506,\n",
       "    'NDCG': 0.17871793876956016,\n",
       "    'F1': 0.1889124132021179,\n",
       "    'HIT_RATE': 2.3904937953171124,\n",
       "    'ARHR': 0.7987475656871302,\n",
       "    'NOVELTY': 0.008635668904812908,\n",
       "    'AVERAGE_POPULARITY': 0.5317611696022949,\n",
       "    'DIVERSITY_MEAN_INTER_LIST': 0.8977607213246102,\n",
       "    'DIVERSITY_HERFINDAHL': 0.989774785684289,\n",
       "    'COVERAGE_ITEM': 0.19295946072465125,\n",
       "    'COVERAGE_ITEM_CORRECT': 0.10233124239303436,\n",
       "    'COVERAGE_USER': 0.9986834196742895,\n",
       "    'COVERAGE_USER_CORRECT': 0.8085377371991185,\n",
       "    'DIVERSITY_GINI': 0.012406631334535961,\n",
       "    'SHANNON_ENTROPY': 7.350796174445975}},\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None],\n",
       " 'result_on_test_best': {'10': {'ROC_AUC': 0.4700438449245519,\n",
       "   'PRECISION': 0.23904937953175287,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.27495851262295795,\n",
       "   'RECALL': 0.15616019154465238,\n",
       "   'MAP': 0.15665723144248708,\n",
       "   'MRR': 0.48063485694814506,\n",
       "   'NDCG': 0.17871793876956016,\n",
       "   'F1': 0.1889124132021179,\n",
       "   'HIT_RATE': 2.3904937953171124,\n",
       "   'ARHR': 0.7987475656871302,\n",
       "   'NOVELTY': 0.008635668904812908,\n",
       "   'AVERAGE_POPULARITY': 0.5317611696022949,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.8977607213246102,\n",
       "   'DIVERSITY_HERFINDAHL': 0.989774785684289,\n",
       "   'COVERAGE_ITEM': 0.19295946072465125,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.10233124239303436,\n",
       "   'COVERAGE_USER': 0.9986834196742895,\n",
       "   'COVERAGE_USER_CORRECT': 0.8085377371991185,\n",
       "   'DIVERSITY_GINI': 0.012406631334535961,\n",
       "   'SHANNON_ENTROPY': 7.350796174445975}},\n",
       " 'time_on_train_list': [11.689192533493042,\n",
       "  11.396177053451538,\n",
       "  11.963291645050049,\n",
       "  12.065917015075684,\n",
       "  12.656835556030273,\n",
       "  12.341151237487793,\n",
       "  12.151384115219116,\n",
       "  13.071793794631958,\n",
       "  11.854823112487793,\n",
       "  12.528700828552246],\n",
       " 'time_on_train_total': 121.71926689147949,\n",
       " 'time_on_train_avg': 12.171926689147949,\n",
       " 'time_on_validation_list': [68.05374026298523,\n",
       "  61.933391094207764,\n",
       "  61.51397776603699,\n",
       "  35.06180739402771,\n",
       "  95.09618592262268,\n",
       "  70.29278826713562,\n",
       "  119.93625807762146,\n",
       "  80.02937054634094,\n",
       "  62.02192687988281,\n",
       "  83.54491543769836],\n",
       " 'time_on_validation_total': 737.4843616485596,\n",
       " 'time_on_validation_avg': 73.74843616485596,\n",
       " 'time_on_test_list': [69.21008539199829,\n",
       "  65.63224411010742,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None,\n",
       "  None],\n",
       " 'time_on_test_total': 134.8423295021057,\n",
       " 'time_on_test_avg': 67.42116475105286,\n",
       " 'result_on_last': {'10': {'ROC_AUC': 0.5334465928491546,\n",
       "   'PRECISION': 0.305380735391055,\n",
       "   'PRECISION_RECALL_MIN_DEN': 0.34610293873470677,\n",
       "   'RECALL': 0.18482762605807215,\n",
       "   'MAP': 0.2432645075760898,\n",
       "   'MRR': 0.5859145692113457,\n",
       "   'NDCG': 0.22233536528638775,\n",
       "   'F1': 0.2302808389450718,\n",
       "   'HIT_RATE': 3.0538073539105266,\n",
       "   'ARHR': 1.0804502676889975,\n",
       "   'NOVELTY': 0.008668107818871092,\n",
       "   'AVERAGE_POPULARITY': 0.5181971273808239,\n",
       "   'DIVERSITY_MEAN_INTER_LIST': 0.9076256000234381,\n",
       "   'DIVERSITY_HERFINDAHL': 0.9907612594182724,\n",
       "   'COVERAGE_ITEM': 0.1829416721280779,\n",
       "   'COVERAGE_ITEM_CORRECT': 0.11328527291452112,\n",
       "   'COVERAGE_USER': 0.9986834196742895,\n",
       "   'COVERAGE_USER_CORRECT': 0.827213142906208,\n",
       "   'DIVERSITY_GINI': 0.013167507472345191,\n",
       "   'SHANNON_ENTROPY': 7.468182907974959}},\n",
       " 'time_on_last_train': 16.444225311279297,\n",
       " 'time_on_last_test': 72.33291292190552}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from Base.DataIO import DataIO\n",
    "\n",
    "data_loader = DataIO(folder_path = output_folder_path)\n",
    "search_metadata = data_loader.load_data(recommender_class.RECOMMENDER_NAME + \"_metadata.zip\")\n",
    "\n",
    "search_metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'topK': 789, 'shrink': 259, 'similarity': 'cosine', 'normalize': True},\n",
       " {'topK': 670, 'shrink': 121, 'similarity': 'cosine', 'normalize': True},\n",
       " {'topK': 80, 'shrink': 487, 'similarity': 'cosine', 'normalize': False},\n",
       " {'topK': 5, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True},\n",
       " {'topK': 1000, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True},\n",
       " {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': True},\n",
       " {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': False},\n",
       " {'topK': 710, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True},\n",
       " {'topK': 734, 'shrink': 0, 'similarity': 'cosine', 'normalize': True},\n",
       " {'topK': 769, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hyperparameters_list = search_metadata[\"hyperparameters_list\"]\n",
    "hyperparameters_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ROC_AUC': 0.5334465928491546,\n",
       " 'PRECISION': 0.305380735391055,\n",
       " 'PRECISION_RECALL_MIN_DEN': 0.34610293873470677,\n",
       " 'RECALL': 0.18482762605807215,\n",
       " 'MAP': 0.2432645075760898,\n",
       " 'MRR': 0.5859145692113457,\n",
       " 'NDCG': 0.22233536528638775,\n",
       " 'F1': 0.2302808389450718,\n",
       " 'HIT_RATE': 3.0538073539105266,\n",
       " 'ARHR': 1.0804502676889975,\n",
       " 'NOVELTY': 0.008668107818871092,\n",
       " 'AVERAGE_POPULARITY': 0.5181971273808239,\n",
       " 'DIVERSITY_MEAN_INTER_LIST': 0.9076256000234381,\n",
       " 'DIVERSITY_HERFINDAHL': 0.9907612594182724,\n",
       " 'COVERAGE_ITEM': 0.1829416721280779,\n",
       " 'COVERAGE_ITEM_CORRECT': 0.11328527291452112,\n",
       " 'COVERAGE_USER': 0.9986834196742895,\n",
       " 'COVERAGE_USER_CORRECT': 0.827213142906208,\n",
       " 'DIVERSITY_GINI': 0.013167507472345191,\n",
       " 'SHANNON_ENTROPY': 7.468182907974959}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_best_on_test = search_metadata[\"result_on_last\"]\n",
    "result_best_on_test[\"10\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'topK': 670, 'shrink': 121, 'similarity': 'cosine', 'normalize': True}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_parameters = search_metadata[\"hyperparameters_best\"]\n",
    "best_parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An example with earlystopping, for BPRMF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters_range_dictionary = {}\n",
    "hyperparameters_range_dictionary[\"sgd_mode\"] = Categorical([\"sgd\", \"adagrad\", \"adam\"])\n",
    "hyperparameters_range_dictionary[\"epochs\"] = Categorical([1500])\n",
    "hyperparameters_range_dictionary[\"num_factors\"] = Integer(1, 200)\n",
    "hyperparameters_range_dictionary[\"batch_size\"] = Categorical([1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024])\n",
    "hyperparameters_range_dictionary[\"positive_reg\"] = Real(low = 1e-5, high = 1e-2, prior = 'log-uniform')\n",
    "hyperparameters_range_dictionary[\"negative_reg\"] = Real(low = 1e-5, high = 1e-2, prior = 'log-uniform')\n",
    "hyperparameters_range_dictionary[\"learning_rate\"] = Real(low = 1e-4, high = 1e-1, prior = 'log-uniform')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "earlystopping_keywargs = {\"validation_every_n\": 5,\n",
    "                          \"stop_on_validation\": True,\n",
    "                          \"evaluator_object\": evaluator_validation,\n",
    "                          \"lower_validations_allowed\": 5,\n",
    "                          \"validation_metric\": metric_to_optimize,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "recommender_input_args = SearchInputRecommenderArgs(\n",
    "    CONSTRUCTOR_POSITIONAL_ARGS = [URM_train],     # For a CBF model simply put [URM_train, ICM_train]\n",
    "    CONSTRUCTOR_KEYWORD_ARGS = {},\n",
    "    FIT_POSITIONAL_ARGS = [],\n",
    "    FIT_KEYWORD_ARGS = earlystopping_keywargs\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "recommender_input_args_last_test = SearchInputRecommenderArgs(\n",
    "    CONSTRUCTOR_POSITIONAL_ARGS = [URM_train_validation],     # For a CBF model simply put [URM_train_validation, ICM_train]\n",
    "    CONSTRUCTOR_KEYWORD_ARGS = {},\n",
    "    FIT_POSITIONAL_ARGS = [],\n",
    "    FIT_KEYWORD_ARGS = earlystopping_keywargs\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_cases = 5\n",
    "n_random_starts = int(n_cases*0.3)\n",
    "metric_to_optimize = \"MAP\"   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from MatrixFactorization.Cython.MatrixFactorization_Cython import MatrixFactorization_BPR_Cython\n",
    "\n",
    "recommender_class = MatrixFactorization_BPR_Cython\n",
    "\n",
    "parameterSearch = SearchBayesianSkopt(recommender_class,\n",
    "                                 evaluator_validation=evaluator_validation,\n",
    "                                 evaluator_test=evaluator_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "MatrixFactorization_BPR_Cython_Recommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'sgd_mode': 'adam', 'epochs': 1500, 'num_factors': 9, 'batch_size': 256, 'positive_reg': 0.0009557358739689882, 'negative_reg': 0.0013254069621555342, 'learning_rate': 0.025468411698118695}\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.87 seconds. BPR loss 8.12E-03. Sample per second: 79753\n",
      "MF_BPR: Epoch 1 of 1500. Elapsed time 0.28 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 1.14 seconds. BPR loss 7.97E-02. Sample per second: 61091\n",
      "MF_BPR: Epoch 2 of 1500. Elapsed time 0.54 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.41 seconds. BPR loss 1.13E+00. Sample per second: 167902\n",
      "MF_BPR: Epoch 3 of 1500. Elapsed time 0.82 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.69 seconds. BPR loss 6.03E+00. Sample per second: 101233\n",
      "MF_BPR: Epoch 4 of 1500. Elapsed time 1.09 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.97 seconds. BPR loss 1.59E+01. Sample per second: 72006\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.24 sec. Users per second: 1753\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.60 sec. Users per second: 1758\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1634017, PRECISION: 0.0545186, PRECISION_RECALL_MIN_DEN: 0.0691154, RECALL: 0.0455113, MAP: 0.0289832, MRR: 0.1270495, NDCG: 0.0401101, F1: 0.0496094, HIT_RATE: 0.5451856, ARHR: 0.1704036, NOVELTY: 0.0107827, AVERAGE_POPULARITY: 0.3288147, DIVERSITY_MEAN_INTER_LIST: 0.9549722, DIVERSITY_HERFINDAHL: 0.9954959, COVERAGE_ITEM: 0.2868645, COVERAGE_ITEM_CORRECT: 0.0584215, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3055897, DIVERSITY_GINI: 0.0471629, SHANNON_ENTROPY: 9.0656596, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 5 of 1500. Elapsed time 40.99 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.86 seconds. BPR loss 2.90E+01. Sample per second: 81368\n",
      "MF_BPR: Epoch 6 of 1500. Elapsed time 41.26 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 1.11 seconds. BPR loss 4.28E+01. Sample per second: 62490\n",
      "MF_BPR: Epoch 7 of 1500. Elapsed time 41.52 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.38 seconds. BPR loss 5.52E+01. Sample per second: 181327\n",
      "MF_BPR: Epoch 8 of 1500. Elapsed time 41.79 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.64 seconds. BPR loss 6.64E+01. Sample per second: 107996\n",
      "MF_BPR: Epoch 9 of 1500. Elapsed time 42.05 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.91 seconds. BPR loss 7.49E+01. Sample per second: 76625\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.17 sec. Users per second: 1757\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.42 sec. Users per second: 1767\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1421010, PRECISION: 0.0446959, PRECISION_RECALL_MIN_DEN: 0.0557717, RECALL: 0.0353801, MAP: 0.0225863, MRR: 0.1076784, NDCG: 0.0325490, F1: 0.0394961, HIT_RATE: 0.4469591, ARHR: 0.1398478, NOVELTY: 0.0114325, AVERAGE_POPULARITY: 0.2639913, DIVERSITY_MEAN_INTER_LIST: 0.9553226, DIVERSITY_HERFINDAHL: 0.9955309, COVERAGE_ITEM: 0.3356427, COVERAGE_ITEM_CORRECT: 0.0571108, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2674089, DIVERSITY_GINI: 0.0575136, SHANNON_ENTROPY: 9.2784362, \n",
      "\n",
      "MF_BPR: Epoch 10 of 1500. Elapsed time 1.36 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.60 seconds. BPR loss 8.34E+01. Sample per second: 116841\n",
      "MF_BPR: Epoch 11 of 1500. Elapsed time 1.37 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.85 seconds. BPR loss 8.87E+01. Sample per second: 82360\n",
      "MF_BPR: Epoch 12 of 1500. Elapsed time 1.37 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 1.10 seconds. BPR loss 9.19E+01. Sample per second: 63232\n",
      "MF_BPR: Epoch 13 of 1500. Elapsed time 1.38 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.37 seconds. BPR loss 9.50E+01. Sample per second: 188225\n",
      "MF_BPR: Epoch 14 of 1500. Elapsed time 1.38 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.63 seconds. BPR loss 9.82E+01. Sample per second: 110885\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.40 sec. Users per second: 1743\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.67 sec. Users per second: 1756\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1436609, PRECISION: 0.0444977, PRECISION_RECALL_MIN_DEN: 0.0552530, RECALL: 0.0350040, MAP: 0.0229698, MRR: 0.1107254, NDCG: 0.0329801, F1: 0.0391840, HIT_RATE: 0.4449774, ARHR: 0.1437265, NOVELTY: 0.0116705, AVERAGE_POPULARITY: 0.2598068, DIVERSITY_MEAN_INTER_LIST: 0.9524973, DIVERSITY_HERFINDAHL: 0.9952484, COVERAGE_ITEM: 0.3425709, COVERAGE_ITEM_CORRECT: 0.0611366, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2640173, DIVERSITY_GINI: 0.0622665, SHANNON_ENTROPY: 9.3589169, \n",
      "\n",
      "MF_BPR: Epoch 15 of 1500. Elapsed time 2.05 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.56 seconds. BPR loss 9.97E+01. Sample per second: 124777\n",
      "MF_BPR: Epoch 16 of 1500. Elapsed time 2.05 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.82 seconds. BPR loss 1.01E+02. Sample per second: 85235\n",
      "MF_BPR: Epoch 17 of 1500. Elapsed time 2.05 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 1.07 seconds. BPR loss 1.01E+02. Sample per second: 64901\n",
      "MF_BPR: Epoch 18 of 1500. Elapsed time 2.06 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.35 seconds. BPR loss 1.01E+02. Sample per second: 201152\n",
      "MF_BPR: Epoch 19 of 1500. Elapsed time 2.06 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.60 seconds. BPR loss 1.02E+02. Sample per second: 115640\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.46 sec. Users per second: 1773\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.49 sec. Users per second: 1764\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1306663, PRECISION: 0.0423652, PRECISION_RECALL_MIN_DEN: 0.0529952, RECALL: 0.0340569, MAP: 0.0210138, MRR: 0.0986704, NDCG: 0.0308020, F1: 0.0377594, HIT_RATE: 0.4236519, ARHR: 0.1293615, NOVELTY: 0.0118756, AVERAGE_POPULARITY: 0.2470838, DIVERSITY_MEAN_INTER_LIST: 0.9620772, DIVERSITY_HERFINDAHL: 0.9962063, COVERAGE_ITEM: 0.3348937, COVERAGE_ITEM_CORRECT: 0.0589832, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2506225, DIVERSITY_GINI: 0.0657597, SHANNON_ENTROPY: 9.5409936, \n",
      "\n",
      "MF_BPR: Epoch 20 of 1500. Elapsed time 2.73 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.36 seconds. BPR loss 1.02E+02. Sample per second: 195525\n",
      "MF_BPR: Epoch 21 of 1500. Elapsed time 2.73 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.62 seconds. BPR loss 1.01E+02. Sample per second: 112965\n",
      "MF_BPR: Epoch 22 of 1500. Elapsed time 2.73 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.89 seconds. BPR loss 1.01E+02. Sample per second: 78240\n",
      "MF_BPR: Epoch 23 of 1500. Elapsed time 2.74 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 1.14 seconds. BPR loss 1.01E+02. Sample per second: 60827\n",
      "MF_BPR: Epoch 24 of 1500. Elapsed time 2.74 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.41 seconds. BPR loss 1.01E+02. Sample per second: 170877\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.52 sec. Users per second: 1769\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.48 sec. Users per second: 1764\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1447711, PRECISION: 0.0471300, PRECISION_RECALL_MIN_DEN: 0.0587848, RECALL: 0.0375028, MAP: 0.0243890, MRR: 0.1123232, NDCG: 0.0351244, F1: 0.0417689, HIT_RATE: 0.4713004, ARHR: 0.1489442, NOVELTY: 0.0118779, AVERAGE_POPULARITY: 0.2645067, DIVERSITY_MEAN_INTER_LIST: 0.9574060, DIVERSITY_HERFINDAHL: 0.9957392, COVERAGE_ITEM: 0.3259994, COVERAGE_ITEM_CORRECT: 0.0560809, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2704142, DIVERSITY_GINI: 0.0602590, SHANNON_ENTROPY: 9.3932527, \n",
      "\n",
      "MF_BPR: Epoch 25 of 1500. Elapsed time 3.41 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 1.16 seconds. BPR loss 1.00E+02. Sample per second: 59873\n",
      "MF_BPR: Epoch 26 of 1500. Elapsed time 3.41 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.42 seconds. BPR loss 9.96E+01. Sample per second: 166070\n",
      "MF_BPR: Epoch 27 of 1500. Elapsed time 3.41 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.69 seconds. BPR loss 9.96E+01. Sample per second: 101647\n",
      "MF_BPR: Epoch 28 of 1500. Elapsed time 3.42 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 0.94 seconds. BPR loss 1.00E+02. Sample per second: 74191\n",
      "MF_BPR: Epoch 29 of 1500. Elapsed time 3.42 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69632 ( 99.63% ) in 1.21 seconds. BPR loss 9.92E+01. Sample per second: 57582\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.22 sec. Users per second: 1721\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.17 sec. Users per second: 1734\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1537910, PRECISION: 0.0504086, PRECISION_RECALL_MIN_DEN: 0.0639268, RECALL: 0.0418918, MAP: 0.0275932, MRR: 0.1224930, NDCG: 0.0394437, F1: 0.0457572, HIT_RATE: 0.5040856, ARHR: 0.1637317, NOVELTY: 0.0119237, AVERAGE_POPULARITY: 0.2671998, DIVERSITY_MEAN_INTER_LIST: 0.9582472, DIVERSITY_HERFINDAHL: 0.9958233, COVERAGE_ITEM: 0.3268421, COVERAGE_ITEM_CORRECT: 0.0552383, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2803171, DIVERSITY_GINI: 0.0609791, SHANNON_ENTROPY: 9.4078383, \n",
      "\n",
      "MF_BPR: Convergence reached! Terminating at epoch 30. Best value for 'MAP' at epoch 5 is 0.0290. Elapsed time 4.10 min\n",
      "MF_BPR: Epoch 30 of 1500. Elapsed time 4.10 min\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.22 sec. Users per second: 1654\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.67 sec. Users per second: 1671\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'sgd_mode': 'adam', 'epochs': 5, 'num_factors': 9, 'batch_size': 256, 'positive_reg': 0.0009557358739689882, 'negative_reg': 0.0013254069621555342, 'learning_rate': 0.025468411698118695} - results: ROC_AUC: 0.1634017, PRECISION: 0.0545186, PRECISION_RECALL_MIN_DEN: 0.0691154, RECALL: 0.0455113, MAP: 0.0289832, MRR: 0.1270495, NDCG: 0.0401101, F1: 0.0496094, HIT_RATE: 0.5451856, ARHR: 0.1704036, NOVELTY: 0.0107827, AVERAGE_POPULARITY: 0.3288147, DIVERSITY_MEAN_INTER_LIST: 0.9549722, DIVERSITY_HERFINDAHL: 0.9954959, COVERAGE_ITEM: 0.2868645, COVERAGE_ITEM_CORRECT: 0.0584215, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3055897, DIVERSITY_GINI: 0.0471629, SHANNON_ENTROPY: 9.0656596, \n",
      "\n",
      "EvaluatorHoldout: Processed 51000 ( 73.08% ) in 30.44 sec. Users per second: 1675\n",
      "EvaluatorHoldout: Processed 69786 ( 100.00% ) in 41.60 sec. Users per second: 1677\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'sgd_mode': 'adam', 'epochs': 5, 'num_factors': 9, 'batch_size': 256, 'positive_reg': 0.0009557358739689882, 'negative_reg': 0.0013254069621555342, 'learning_rate': 0.025468411698118695} - results:\n",
      "CUTOFF: 10 - ROC_AUC: 0.1805530, PRECISION: 0.0676683, PRECISION_RECALL_MIN_DEN: 0.0786002, RECALL: 0.0452358, MAP: 0.0351252, MRR: 0.1491072, NDCG: 0.0427381, F1: 0.0542235, HIT_RATE: 0.6766830, ARHR: 0.2110348, NOVELTY: 0.0107820, AVERAGE_POPULARITY: 0.3288060, DIVERSITY_MEAN_INTER_LIST: 0.9549708, DIVERSITY_HERFINDAHL: 0.9954957, COVERAGE_ITEM: 0.2869581, COVERAGE_ITEM_CORRECT: 0.0634772, COVERAGE_USER: 0.9986834, COVERAGE_USER_CORRECT: 0.3413664, DIVERSITY_GINI: 0.0471529, SHANNON_ENTROPY: 9.0653945, \n",
      "\n",
      "\n",
      "DataIO: Json dumps supports only 'str' as dictionary keys. Transforming keys to string, note that this will alter the mapper content.\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 329.8145\n",
      "Function value obtained: -0.0290\n",
      "Current minimum: -0.0290\n",
      "Iteration No: 2 started. Searching for the next optimal point.\n",
      "MatrixFactorization_BPR_Cython_Recommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'sgd_mode': 'adagrad', 'epochs': 1500, 'num_factors': 200, 'batch_size': 64, 'positive_reg': 1e-05, 'negative_reg': 1e-05, 'learning_rate': 0.0001}\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 5.34 seconds. BPR loss 3.97E-02. Sample per second: 13065\n",
      "MF_BPR: Epoch 1 of 1500. Elapsed time 4.36 sec\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.35 seconds. BPR loss 3.98E-02. Sample per second: 16038\n",
      "MF_BPR: Epoch 2 of 1500. Elapsed time 8.37 sec\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.32 seconds. BPR loss 4.00E-02. Sample per second: 16153\n",
      "MF_BPR: Epoch 3 of 1500. Elapsed time 12.34 sec\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.27 seconds. BPR loss 4.00E-02. Sample per second: 16334\n",
      "MF_BPR: Epoch 4 of 1500. Elapsed time 16.29 sec\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.41 seconds. BPR loss 3.99E-02. Sample per second: 15819\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.44 sec. Users per second: 1807\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.34 sec. Users per second: 1770\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0102341, PRECISION: 0.0021469, PRECISION_RECALL_MIN_DEN: 0.0023336, RECALL: 0.0009103, MAP: 0.0006721, MRR: 0.0059730, NDCG: 0.0008587, F1: 0.0012785, HIT_RATE: 0.0214691, ARHR: 0.0060739, NOVELTY: 0.0149519, AVERAGE_POPULARITY: 0.0248600, DIVERSITY_MEAN_INTER_LIST: 0.9987952, DIVERSITY_HERFINDAHL: 0.9998781, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0923135, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0207648, DIVERSITY_GINI: 0.7029518, SHANNON_ENTROPY: 13.1779494, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 5 of 1500. Elapsed time 59.88 sec\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.99 seconds. BPR loss 4.00E-02. Sample per second: 13990\n",
      "MF_BPR: Epoch 6 of 1500. Elapsed time 1.07 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 5.26 seconds. BPR loss 3.99E-02. Sample per second: 13273\n",
      "MF_BPR: Epoch 7 of 1500. Elapsed time 1.14 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.44 seconds. BPR loss 3.98E-02. Sample per second: 15716\n",
      "MF_BPR: Epoch 8 of 1500. Elapsed time 1.21 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.59 seconds. BPR loss 4.05E-02. Sample per second: 15204\n",
      "MF_BPR: Epoch 9 of 1500. Elapsed time 1.28 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.71 seconds. BPR loss 4.00E-02. Sample per second: 14832\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.17 sec. Users per second: 1624\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.35 sec. Users per second: 1644\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0102149, PRECISION: 0.0021455, PRECISION_RECALL_MIN_DEN: 0.0023322, RECALL: 0.0009094, MAP: 0.0006713, MRR: 0.0059655, NDCG: 0.0008571, F1: 0.0012774, HIT_RATE: 0.0214547, ARHR: 0.0060664, NOVELTY: 0.0149520, AVERAGE_POPULARITY: 0.0248519, DIVERSITY_MEAN_INTER_LIST: 0.9987952, DIVERSITY_HERFINDAHL: 0.9998781, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0922198, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0207505, DIVERSITY_GINI: 0.7029390, SHANNON_ENTROPY: 13.1779440, \n",
      "\n",
      "MF_BPR: Epoch 10 of 1500. Elapsed time 2.05 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.09 seconds. BPR loss 3.99E-02. Sample per second: 17082\n",
      "MF_BPR: Epoch 11 of 1500. Elapsed time 2.12 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.08 seconds. BPR loss 3.98E-02. Sample per second: 17111\n",
      "MF_BPR: Epoch 12 of 1500. Elapsed time 2.18 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.03 seconds. BPR loss 4.00E-02. Sample per second: 17308\n",
      "MF_BPR: Epoch 13 of 1500. Elapsed time 2.25 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 3.98 seconds. BPR loss 3.96E-02. Sample per second: 17528\n",
      "MF_BPR: Epoch 14 of 1500. Elapsed time 2.32 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.92 seconds. BPR loss 3.95E-02. Sample per second: 14179\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.18 sec. Users per second: 1690\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.81 sec. Users per second: 1666\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0102245, PRECISION: 0.0021498, PRECISION_RECALL_MIN_DEN: 0.0023375, RECALL: 0.0009121, MAP: 0.0006717, MRR: 0.0059662, NDCG: 0.0008594, F1: 0.0012808, HIT_RATE: 0.0214978, ARHR: 0.0060685, NOVELTY: 0.0149522, AVERAGE_POPULARITY: 0.0248404, DIVERSITY_MEAN_INTER_LIST: 0.9987952, DIVERSITY_HERFINDAHL: 0.9998781, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0923135, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0207791, DIVERSITY_GINI: 0.7029237, SHANNON_ENTROPY: 13.1779151, \n",
      "\n",
      "MF_BPR: Epoch 15 of 1500. Elapsed time 3.08 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.97 seconds. BPR loss 3.94E-02. Sample per second: 14051\n",
      "MF_BPR: Epoch 16 of 1500. Elapsed time 3.15 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 5.28 seconds. BPR loss 3.96E-02. Sample per second: 13218\n",
      "MF_BPR: Epoch 17 of 1500. Elapsed time 3.22 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.26 seconds. BPR loss 3.96E-02. Sample per second: 16376\n",
      "MF_BPR: Epoch 18 of 1500. Elapsed time 3.29 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.26 seconds. BPR loss 3.98E-02. Sample per second: 16400\n",
      "MF_BPR: Epoch 19 of 1500. Elapsed time 3.35 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.28 seconds. BPR loss 3.98E-02. Sample per second: 16310\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.29 sec. Users per second: 1618\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.77 sec. Users per second: 1591\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0102222, PRECISION: 0.0021469, PRECISION_RECALL_MIN_DEN: 0.0023346, RECALL: 0.0009105, MAP: 0.0006710, MRR: 0.0059604, NDCG: 0.0008586, F1: 0.0012787, HIT_RATE: 0.0214691, ARHR: 0.0060615, NOVELTY: 0.0149522, AVERAGE_POPULARITY: 0.0248332, DIVERSITY_MEAN_INTER_LIST: 0.9987953, DIVERSITY_HERFINDAHL: 0.9998781, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0924071, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0207648, DIVERSITY_GINI: 0.7029924, SHANNON_ENTROPY: 13.1780043, \n",
      "\n",
      "MF_BPR: Epoch 20 of 1500. Elapsed time 4.15 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.44 seconds. BPR loss 4.04E-02. Sample per second: 15721\n",
      "MF_BPR: Epoch 21 of 1500. Elapsed time 4.22 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.37 seconds. BPR loss 3.99E-02. Sample per second: 15976\n",
      "MF_BPR: Epoch 22 of 1500. Elapsed time 4.29 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.31 seconds. BPR loss 4.01E-02. Sample per second: 16187\n",
      "MF_BPR: Epoch 23 of 1500. Elapsed time 4.36 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.29 seconds. BPR loss 4.01E-02. Sample per second: 16266\n",
      "MF_BPR: Epoch 24 of 1500. Elapsed time 4.42 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.22 seconds. BPR loss 3.96E-02. Sample per second: 16543\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.51 sec. Users per second: 1705\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.02 sec. Users per second: 1698\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0102037, PRECISION: 0.0021455, PRECISION_RECALL_MIN_DEN: 0.0023332, RECALL: 0.0009111, MAP: 0.0006704, MRR: 0.0059551, NDCG: 0.0008586, F1: 0.0012790, HIT_RATE: 0.0214547, ARHR: 0.0060559, NOVELTY: 0.0149524, AVERAGE_POPULARITY: 0.0248260, DIVERSITY_MEAN_INTER_LIST: 0.9987952, DIVERSITY_HERFINDAHL: 0.9998781, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0923135, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0207505, DIVERSITY_GINI: 0.7029397, SHANNON_ENTROPY: 13.1779463, \n",
      "\n",
      "MF_BPR: Epoch 25 of 1500. Elapsed time 5.17 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.39 seconds. BPR loss 3.98E-02. Sample per second: 15893\n",
      "MF_BPR: Epoch 26 of 1500. Elapsed time 5.24 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.36 seconds. BPR loss 4.00E-02. Sample per second: 16009\n",
      "MF_BPR: Epoch 27 of 1500. Elapsed time 5.31 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.38 seconds. BPR loss 4.02E-02. Sample per second: 15951\n",
      "MF_BPR: Epoch 28 of 1500. Elapsed time 5.37 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.31 seconds. BPR loss 4.00E-02. Sample per second: 16190\n",
      "MF_BPR: Epoch 29 of 1500. Elapsed time 5.44 min\n",
      "MF_BPR: Processed 69824 ( 99.91% ) in 4.25 seconds. BPR loss 3.98E-02. Sample per second: 16412\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.16 sec. Users per second: 1691\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.02 sec. Users per second: 1698\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0102006, PRECISION: 0.0021440, PRECISION_RECALL_MIN_DEN: 0.0023317, RECALL: 0.0009108, MAP: 0.0006702, MRR: 0.0059531, NDCG: 0.0008578, F1: 0.0012785, HIT_RATE: 0.0214404, ARHR: 0.0060540, NOVELTY: 0.0149524, AVERAGE_POPULARITY: 0.0248212, DIVERSITY_MEAN_INTER_LIST: 0.9987952, DIVERSITY_HERFINDAHL: 0.9998781, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0924071, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0207361, DIVERSITY_GINI: 0.7029195, SHANNON_ENTROPY: 13.1779191, \n",
      "\n",
      "MF_BPR: Convergence reached! Terminating at epoch 30. Best value for 'MAP' at epoch 5 is 0.0007. Elapsed time 6.19 min\n",
      "MF_BPR: Epoch 30 of 1500. Elapsed time 6.19 min\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.42 sec. Users per second: 1644\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.71 sec. Users per second: 1630\n",
      "SearchBayesianSkopt: Config 1 is suboptimal. Config: {'sgd_mode': 'adagrad', 'epochs': 5, 'num_factors': 200, 'batch_size': 64, 'positive_reg': 1e-05, 'negative_reg': 1e-05, 'learning_rate': 0.0001} - results: ROC_AUC: 0.0102341, PRECISION: 0.0021469, PRECISION_RECALL_MIN_DEN: 0.0023336, RECALL: 0.0009103, MAP: 0.0006721, MRR: 0.0059730, NDCG: 0.0008587, F1: 0.0012785, HIT_RATE: 0.0214691, ARHR: 0.0060739, NOVELTY: 0.0149519, AVERAGE_POPULARITY: 0.0248600, DIVERSITY_MEAN_INTER_LIST: 0.9987952, DIVERSITY_HERFINDAHL: 0.9998781, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0923135, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0207648, DIVERSITY_GINI: 0.7029518, SHANNON_ENTROPY: 13.1779494, \n",
      "\n",
      "Iteration No: 2 ended. Search finished for the next optimal point.\n",
      "Time taken: 415.8438\n",
      "Function value obtained: -0.0007\n",
      "Current minimum: -0.0290\n",
      "Iteration No: 3 started. Searching for the next optimal point.\n",
      "MatrixFactorization_BPR_Cython_Recommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'sgd_mode': 'sgd', 'epochs': 1500, 'num_factors': 1, 'batch_size': 16, 'positive_reg': 0.01, 'negative_reg': 0.01, 'learning_rate': 0.1}\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.16 seconds. BPR loss 1.90E-04. Sample per second: 445456\n",
      "MF_BPR: Epoch 1 of 1500. Elapsed time 0.05 sec\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.20 seconds. BPR loss 1.89E-04. Sample per second: 349661\n",
      "MF_BPR: Epoch 2 of 1500. Elapsed time 0.09 sec\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.25 seconds. BPR loss 1.88E-04. Sample per second: 280881\n",
      "MF_BPR: Epoch 3 of 1500. Elapsed time 0.14 sec\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.29 seconds. BPR loss 1.88E-04. Sample per second: 240345\n",
      "MF_BPR: Epoch 4 of 1500. Elapsed time 0.18 sec\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.33 seconds. BPR loss 1.87E-04. Sample per second: 208778\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.48 sec. Users per second: 1706\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.21 sec. Users per second: 1732\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0026350, PRECISION: 0.0006994, PRECISION_RECALL_MIN_DEN: 0.0007122, RECALL: 0.0001475, MAP: 0.0001481, MRR: 0.0013954, NDCG: 0.0001294, F1: 0.0002437, HIT_RATE: 0.0069936, ARHR: 0.0014228, NOVELTY: 0.0158104, AVERAGE_POPULARITY: 0.0085894, DIVERSITY_MEAN_INTER_LIST: 0.5027165, DIVERSITY_HERFINDAHL: 0.9502709, COVERAGE_ITEM: 0.0030896, COVERAGE_ITEM_CORRECT: 0.0023406, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0067546, DIVERSITY_GINI: 0.0019652, SHANNON_ENTROPY: 4.3420170, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 5 of 1500. Elapsed time 40.44 sec\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.59 seconds. BPR loss 1.85E-04. Sample per second: 118383\n",
      "MF_BPR: Epoch 6 of 1500. Elapsed time 40.48 sec\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.63 seconds. BPR loss 1.85E-04. Sample per second: 110178\n",
      "MF_BPR: Epoch 7 of 1500. Elapsed time 40.52 sec\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.68 seconds. BPR loss 1.83E-04. Sample per second: 103186\n",
      "MF_BPR: Epoch 8 of 1500. Elapsed time 40.57 sec\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.72 seconds. BPR loss 1.83E-04. Sample per second: 96765\n",
      "MF_BPR: Epoch 9 of 1500. Elapsed time 40.61 sec\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.77 seconds. BPR loss 1.79E-04. Sample per second: 90975\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.25 sec. Users per second: 1686\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.05 sec. Users per second: 1696\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0026074, PRECISION: 0.0005644, PRECISION_RECALL_MIN_DEN: 0.0005751, RECALL: 0.0001142, MAP: 0.0001321, MRR: 0.0012450, NDCG: 0.0001158, F1: 0.0001900, HIT_RATE: 0.0056437, ARHR: 0.0012688, NOVELTY: 0.0160049, AVERAGE_POPULARITY: 0.0066202, DIVERSITY_MEAN_INTER_LIST: 0.5020943, DIVERSITY_HERFINDAHL: 0.9502087, COVERAGE_ITEM: 0.0030896, COVERAGE_ITEM_CORRECT: 0.0022470, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0054380, DIVERSITY_GINI: 0.0019637, SHANNON_ENTROPY: 4.3383380, \n",
      "\n",
      "MF_BPR: Epoch 10 of 1500. Elapsed time 1.36 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.88 seconds. BPR loss 1.81E-04. Sample per second: 79744\n",
      "MF_BPR: Epoch 11 of 1500. Elapsed time 1.36 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.92 seconds. BPR loss 1.80E-04. Sample per second: 76100\n",
      "MF_BPR: Epoch 12 of 1500. Elapsed time 1.36 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.96 seconds. BPR loss 1.76E-04. Sample per second: 72623\n",
      "MF_BPR: Epoch 13 of 1500. Elapsed time 1.36 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 1.00 seconds. BPR loss 1.77E-04. Sample per second: 69658\n",
      "MF_BPR: Epoch 14 of 1500. Elapsed time 1.36 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.05 seconds. BPR loss 1.79E-04. Sample per second: 1424717\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.39 sec. Users per second: 1744\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.15 sec. Users per second: 1692\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0026009, PRECISION: 0.0005601, PRECISION_RECALL_MIN_DEN: 0.0005708, RECALL: 0.0001136, MAP: 0.0001317, MRR: 0.0012422, NDCG: 0.0001152, F1: 0.0001889, HIT_RATE: 0.0056006, ARHR: 0.0012654, NOVELTY: 0.0160052, AVERAGE_POPULARITY: 0.0066172, DIVERSITY_MEAN_INTER_LIST: 0.5020935, DIVERSITY_HERFINDAHL: 0.9502086, COVERAGE_ITEM: 0.0029023, COVERAGE_ITEM_CORRECT: 0.0024342, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0053951, DIVERSITY_GINI: 0.0019636, SHANNON_ENTROPY: 4.3382871, \n",
      "\n",
      "MF_BPR: Epoch 15 of 1500. Elapsed time 2.05 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.27 seconds. BPR loss 1.79E-04. Sample per second: 255336\n",
      "MF_BPR: Epoch 16 of 1500. Elapsed time 2.05 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.32 seconds. BPR loss 1.75E-04. Sample per second: 219310\n",
      "MF_BPR: Epoch 17 of 1500. Elapsed time 2.05 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.37 seconds. BPR loss 1.74E-04. Sample per second: 189789\n",
      "MF_BPR: Epoch 18 of 1500. Elapsed time 2.05 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.41 seconds. BPR loss 1.77E-04. Sample per second: 168728\n",
      "MF_BPR: Epoch 19 of 1500. Elapsed time 2.06 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.47 seconds. BPR loss 1.75E-04. Sample per second: 149923\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.28 sec. Users per second: 1684\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.01 sec. Users per second: 1698\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0027857, PRECISION: 0.0005371, PRECISION_RECALL_MIN_DEN: 0.0005478, RECALL: 0.0001115, MAP: 0.0001328, MRR: 0.0012590, NDCG: 0.0001150, F1: 0.0001846, HIT_RATE: 0.0053709, ARHR: 0.0012789, NOVELTY: 0.0160090, AVERAGE_POPULARITY: 0.0065974, DIVERSITY_MEAN_INTER_LIST: 0.5020902, DIVERSITY_HERFINDAHL: 0.9502083, COVERAGE_ITEM: 0.0029023, COVERAGE_ITEM_CORRECT: 0.0024342, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0051948, DIVERSITY_GINI: 0.0019632, SHANNON_ENTROPY: 4.3380301, \n",
      "\n",
      "MF_BPR: Epoch 20 of 1500. Elapsed time 2.74 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.52 seconds. BPR loss 1.70E-04. Sample per second: 133535\n",
      "MF_BPR: Epoch 21 of 1500. Elapsed time 2.74 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.57 seconds. BPR loss 1.74E-04. Sample per second: 122752\n",
      "MF_BPR: Epoch 22 of 1500. Elapsed time 2.74 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.61 seconds. BPR loss 1.71E-04. Sample per second: 114513\n",
      "MF_BPR: Epoch 23 of 1500. Elapsed time 2.74 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.65 seconds. BPR loss 1.72E-04. Sample per second: 107146\n",
      "MF_BPR: Epoch 24 of 1500. Elapsed time 2.74 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.69 seconds. BPR loss 1.70E-04. Sample per second: 100814\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.24 sec. Users per second: 1753\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.66 sec. Users per second: 1756\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0027809, PRECISION: 0.0005328, PRECISION_RECALL_MIN_DEN: 0.0005435, RECALL: 0.0001112, MAP: 0.0001322, MRR: 0.0012531, NDCG: 0.0001144, F1: 0.0001840, HIT_RATE: 0.0053278, ARHR: 0.0012730, NOVELTY: 0.0160091, AVERAGE_POPULARITY: 0.0065966, DIVERSITY_MEAN_INTER_LIST: 0.5020862, DIVERSITY_HERFINDAHL: 0.9502079, COVERAGE_ITEM: 0.0029023, COVERAGE_ITEM_CORRECT: 0.0023406, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0051518, DIVERSITY_GINI: 0.0019623, SHANNON_ENTROPY: 4.3380170, \n",
      "\n",
      "MF_BPR: Epoch 25 of 1500. Elapsed time 3.40 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.40 seconds. BPR loss 1.69E-04. Sample per second: 172888\n",
      "MF_BPR: Epoch 26 of 1500. Elapsed time 3.40 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.45 seconds. BPR loss 1.70E-04. Sample per second: 156980\n",
      "MF_BPR: Epoch 27 of 1500. Elapsed time 3.41 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.49 seconds. BPR loss 1.69E-04. Sample per second: 144051\n",
      "MF_BPR: Epoch 28 of 1500. Elapsed time 3.41 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.53 seconds. BPR loss 1.68E-04. Sample per second: 133088\n",
      "MF_BPR: Epoch 29 of 1500. Elapsed time 3.41 min\n",
      "MF_BPR: Processed 69872 ( 99.98% ) in 0.57 seconds. BPR loss 1.73E-04. Sample per second: 123456\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.40 sec. Users per second: 1809\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.54 sec. Users per second: 1807\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0028175, PRECISION: 0.0005299, PRECISION_RECALL_MIN_DEN: 0.0005406, RECALL: 0.0001112, MAP: 0.0001329, MRR: 0.0012569, NDCG: 0.0001150, F1: 0.0001838, HIT_RATE: 0.0052991, ARHR: 0.0012783, NOVELTY: 0.0160094, AVERAGE_POPULARITY: 0.0065962, DIVERSITY_MEAN_INTER_LIST: 0.5020778, DIVERSITY_HERFINDAHL: 0.9502071, COVERAGE_ITEM: 0.0029023, COVERAGE_ITEM_CORRECT: 0.0022470, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0051232, DIVERSITY_GINI: 0.0019612, SHANNON_ENTROPY: 4.3379688, \n",
      "\n",
      "MF_BPR: Convergence reached! Terminating at epoch 30. Best value for 'MAP' at epoch 5 is 0.0001. Elapsed time 4.05 min\n",
      "MF_BPR: Epoch 30 of 1500. Elapsed time 4.05 min\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.22 sec. Users per second: 1754\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.83 sec. Users per second: 1748\n",
      "SearchBayesianSkopt: Config 2 is suboptimal. Config: {'sgd_mode': 'sgd', 'epochs': 5, 'num_factors': 1, 'batch_size': 16, 'positive_reg': 0.01, 'negative_reg': 0.01, 'learning_rate': 0.1} - results: ROC_AUC: 0.0026350, PRECISION: 0.0006994, PRECISION_RECALL_MIN_DEN: 0.0007122, RECALL: 0.0001475, MAP: 0.0001481, MRR: 0.0013954, NDCG: 0.0001294, F1: 0.0002437, HIT_RATE: 0.0069936, ARHR: 0.0014228, NOVELTY: 0.0158104, AVERAGE_POPULARITY: 0.0085894, DIVERSITY_MEAN_INTER_LIST: 0.5027165, DIVERSITY_HERFINDAHL: 0.9502709, COVERAGE_ITEM: 0.0030896, COVERAGE_ITEM_CORRECT: 0.0023406, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0067546, DIVERSITY_GINI: 0.0019652, SHANNON_ENTROPY: 4.3420170, \n",
      "\n",
      "Iteration No: 3 ended. Search finished for the next optimal point.\n",
      "Time taken: 283.6390\n",
      "Function value obtained: -0.0001\n",
      "Current minimum: -0.0290\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "MatrixFactorization_BPR_Cython_Recommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'sgd_mode': 'adam', 'epochs': 1500, 'num_factors': 199, 'batch_size': 256, 'positive_reg': 1e-05, 'negative_reg': 1e-05, 'learning_rate': 0.0001}\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.85 seconds. BPR loss 4.00E-02. Sample per second: 14360\n",
      "MF_BPR: Epoch 1 of 1500. Elapsed time 4.41 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.19 seconds. BPR loss 3.99E-02. Sample per second: 13427\n",
      "MF_BPR: Epoch 2 of 1500. Elapsed time 8.75 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.55 seconds. BPR loss 4.00E-02. Sample per second: 15302\n",
      "MF_BPR: Epoch 3 of 1500. Elapsed time 13.11 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.87 seconds. BPR loss 4.00E-02. Sample per second: 14312\n",
      "MF_BPR: Epoch 4 of 1500. Elapsed time 17.43 sec\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.19 seconds. BPR loss 3.98E-02. Sample per second: 13406\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.48 sec. Users per second: 1673\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.74 sec. Users per second: 1669\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0114600, PRECISION: 0.0023250, PRECISION_RECALL_MIN_DEN: 0.0025182, RECALL: 0.0009592, MAP: 0.0007688, MRR: 0.0069034, NDCG: 0.0009312, F1: 0.0013581, HIT_RATE: 0.0232498, ARHR: 0.0070273, NOVELTY: 0.0149612, AVERAGE_POPULARITY: 0.0253054, DIVERSITY_MEAN_INTER_LIST: 0.9987879, DIVERSITY_HERFINDAHL: 0.9998774, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0961520, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0224248, DIVERSITY_GINI: 0.7012928, SHANNON_ENTROPY: 13.1746251, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 5 of 1500. Elapsed time 1.06 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.38 seconds. BPR loss 3.94E-02. Sample per second: 15882\n",
      "MF_BPR: Epoch 6 of 1500. Elapsed time 1.13 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.78 seconds. BPR loss 3.97E-02. Sample per second: 14573\n",
      "MF_BPR: Epoch 7 of 1500. Elapsed time 1.21 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.15 seconds. BPR loss 4.01E-02. Sample per second: 13512\n",
      "MF_BPR: Epoch 8 of 1500. Elapsed time 1.28 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.60 seconds. BPR loss 4.02E-02. Sample per second: 15150\n",
      "MF_BPR: Epoch 9 of 1500. Elapsed time 1.35 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.07 seconds. BPR loss 4.00E-02. Sample per second: 13728\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.27 sec. Users per second: 1619\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.87 sec. Users per second: 1624\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0114197, PRECISION: 0.0023221, PRECISION_RECALL_MIN_DEN: 0.0025072, RECALL: 0.0009449, MAP: 0.0007697, MRR: 0.0068760, NDCG: 0.0009231, F1: 0.0013432, HIT_RATE: 0.0232211, ARHR: 0.0070067, NOVELTY: 0.0149635, AVERAGE_POPULARITY: 0.0251995, DIVERSITY_MEAN_INTER_LIST: 0.9987874, DIVERSITY_HERFINDAHL: 0.9998773, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0957775, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0222960, DIVERSITY_GINI: 0.7009728, SHANNON_ENTROPY: 13.1742517, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 10 of 1500. Elapsed time 2.14 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.40 seconds. BPR loss 3.97E-02. Sample per second: 15825\n",
      "MF_BPR: Epoch 11 of 1500. Elapsed time 2.22 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.73 seconds. BPR loss 3.98E-02. Sample per second: 14728\n",
      "MF_BPR: Epoch 12 of 1500. Elapsed time 2.29 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.62 seconds. BPR loss 3.99E-02. Sample per second: 12389\n",
      "MF_BPR: Epoch 13 of 1500. Elapsed time 2.37 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.89 seconds. BPR loss 3.98E-02. Sample per second: 14242\n",
      "MF_BPR: Epoch 14 of 1500. Elapsed time 2.44 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.24 seconds. BPR loss 4.01E-02. Sample per second: 13283\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.64 sec. Users per second: 1665\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.32 sec. Users per second: 1646\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0114022, PRECISION: 0.0023393, PRECISION_RECALL_MIN_DEN: 0.0025215, RECALL: 0.0009515, MAP: 0.0007687, MRR: 0.0068784, NDCG: 0.0009275, F1: 0.0013527, HIT_RATE: 0.0233934, ARHR: 0.0070115, NOVELTY: 0.0149642, AVERAGE_POPULARITY: 0.0251622, DIVERSITY_MEAN_INTER_LIST: 0.9987870, DIVERSITY_HERFINDAHL: 0.9998773, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0967138, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0224964, DIVERSITY_GINI: 0.7006057, SHANNON_ENTROPY: 13.1737917, \n",
      "\n",
      "MF_BPR: Epoch 15 of 1500. Elapsed time 3.22 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.96 seconds. BPR loss 3.97E-02. Sample per second: 14026\n",
      "MF_BPR: Epoch 16 of 1500. Elapsed time 3.29 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.36 seconds. BPR loss 3.98E-02. Sample per second: 12989\n",
      "MF_BPR: Epoch 17 of 1500. Elapsed time 3.37 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.76 seconds. BPR loss 3.96E-02. Sample per second: 14626\n",
      "MF_BPR: Epoch 18 of 1500. Elapsed time 3.44 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.12 seconds. BPR loss 3.99E-02. Sample per second: 13601\n",
      "MF_BPR: Epoch 19 of 1500. Elapsed time 3.51 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.51 seconds. BPR loss 4.00E-02. Sample per second: 15432\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.93% ) in 30.10 sec. Users per second: 1595\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.44 sec. Users per second: 1603\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0114433, PRECISION: 0.0023508, PRECISION_RECALL_MIN_DEN: 0.0025366, RECALL: 0.0009588, MAP: 0.0007764, MRR: 0.0069322, NDCG: 0.0009330, F1: 0.0013621, HIT_RATE: 0.0235083, ARHR: 0.0070703, NOVELTY: 0.0149613, AVERAGE_POPULARITY: 0.0252812, DIVERSITY_MEAN_INTER_LIST: 0.9987877, DIVERSITY_HERFINDAHL: 0.9998773, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0970883, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0225679, DIVERSITY_GINI: 0.7010557, SHANNON_ENTROPY: 13.1744226, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 20 of 1500. Elapsed time 4.31 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.39 seconds. BPR loss 3.98E-02. Sample per second: 15851\n",
      "MF_BPR: Epoch 21 of 1500. Elapsed time 4.38 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.66 seconds. BPR loss 3.99E-02. Sample per second: 14947\n",
      "MF_BPR: Epoch 22 of 1500. Elapsed time 4.45 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.89 seconds. BPR loss 4.01E-02. Sample per second: 14250\n",
      "MF_BPR: Epoch 23 of 1500. Elapsed time 4.52 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.19 seconds. BPR loss 4.02E-02. Sample per second: 13408\n",
      "MF_BPR: Epoch 24 of 1500. Elapsed time 4.60 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.46 seconds. BPR loss 4.02E-02. Sample per second: 15626\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.38 sec. Users per second: 1711\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.57 sec. Users per second: 1716\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0114913, PRECISION: 0.0023767, PRECISION_RECALL_MIN_DEN: 0.0025694, RECALL: 0.0009787, MAP: 0.0007835, MRR: 0.0069947, NDCG: 0.0009493, F1: 0.0013865, HIT_RATE: 0.0237668, ARHR: 0.0071300, NOVELTY: 0.0149562, AVERAGE_POPULARITY: 0.0254912, DIVERSITY_MEAN_INTER_LIST: 0.9987875, DIVERSITY_HERFINDAHL: 0.9998773, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0965265, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0228398, DIVERSITY_GINI: 0.7010070, SHANNON_ENTROPY: 13.1742859, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 25 of 1500. Elapsed time 5.34 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.35 seconds. BPR loss 4.00E-02. Sample per second: 16021\n",
      "MF_BPR: Epoch 26 of 1500. Elapsed time 5.42 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.55 seconds. BPR loss 4.04E-02. Sample per second: 15316\n",
      "MF_BPR: Epoch 27 of 1500. Elapsed time 5.49 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.78 seconds. BPR loss 4.00E-02. Sample per second: 14553\n",
      "MF_BPR: Epoch 28 of 1500. Elapsed time 5.56 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.01 seconds. BPR loss 3.99E-02. Sample per second: 13892\n",
      "MF_BPR: Epoch 29 of 1500. Elapsed time 5.63 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.28 seconds. BPR loss 4.03E-02. Sample per second: 16267\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.56 sec. Users per second: 1702\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.00 sec. Users per second: 1699\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0116628, PRECISION: 0.0023982, PRECISION_RECALL_MIN_DEN: 0.0025888, RECALL: 0.0009814, MAP: 0.0007976, MRR: 0.0070954, NDCG: 0.0009640, F1: 0.0013928, HIT_RATE: 0.0239822, ARHR: 0.0072332, NOVELTY: 0.0149492, AVERAGE_POPULARITY: 0.0258491, DIVERSITY_MEAN_INTER_LIST: 0.9987875, DIVERSITY_HERFINDAHL: 0.9998773, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0974628, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0230545, DIVERSITY_GINI: 0.7011095, SHANNON_ENTROPY: 13.1743841, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 30 of 1500. Elapsed time 6.38 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.65 seconds. BPR loss 4.07E-02. Sample per second: 14960\n",
      "MF_BPR: Epoch 31 of 1500. Elapsed time 6.45 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.95 seconds. BPR loss 4.01E-02. Sample per second: 14061\n",
      "MF_BPR: Epoch 32 of 1500. Elapsed time 6.53 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.25 seconds. BPR loss 4.05E-02. Sample per second: 13257\n",
      "MF_BPR: Epoch 33 of 1500. Elapsed time 6.60 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.56 seconds. BPR loss 4.10E-02. Sample per second: 15256\n",
      "MF_BPR: Epoch 34 of 1500. Elapsed time 6.67 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.91 seconds. BPR loss 4.10E-02. Sample per second: 14170\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.59 sec. Users per second: 1667\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.69 sec. Users per second: 1631\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0119174, PRECISION: 0.0024212, PRECISION_RECALL_MIN_DEN: 0.0026246, RECALL: 0.0010182, MAP: 0.0008165, MRR: 0.0072628, NDCG: 0.0009983, F1: 0.0014336, HIT_RATE: 0.0242120, ARHR: 0.0074016, NOVELTY: 0.0149398, AVERAGE_POPULARITY: 0.0262747, DIVERSITY_MEAN_INTER_LIST: 0.9987872, DIVERSITY_HERFINDAHL: 0.9998773, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0963393, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0232548, DIVERSITY_GINI: 0.7009967, SHANNON_ENTROPY: 13.1743141, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 35 of 1500. Elapsed time 7.45 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.33 seconds. BPR loss 4.03E-02. Sample per second: 13054\n",
      "MF_BPR: Epoch 36 of 1500. Elapsed time 7.53 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.98 seconds. BPR loss 4.09E-02. Sample per second: 13990\n",
      "MF_BPR: Epoch 37 of 1500. Elapsed time 7.61 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.19 seconds. BPR loss 4.09E-02. Sample per second: 13408\n",
      "MF_BPR: Epoch 38 of 1500. Elapsed time 7.68 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.50 seconds. BPR loss 4.06E-02. Sample per second: 15462\n",
      "MF_BPR: Epoch 39 of 1500. Elapsed time 7.75 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.85 seconds. BPR loss 4.07E-02. Sample per second: 14365\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.48 sec. Users per second: 1673\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.32 sec. Users per second: 1685\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0121983, PRECISION: 0.0024930, PRECISION_RECALL_MIN_DEN: 0.0027129, RECALL: 0.0010715, MAP: 0.0008430, MRR: 0.0074688, NDCG: 0.0010444, F1: 0.0014988, HIT_RATE: 0.0249300, ARHR: 0.0076202, NOVELTY: 0.0149275, AVERAGE_POPULARITY: 0.0268715, DIVERSITY_MEAN_INTER_LIST: 0.9987875, DIVERSITY_HERFINDAHL: 0.9998773, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0969010, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0238702, DIVERSITY_GINI: 0.7012563, SHANNON_ENTROPY: 13.1746066, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 40 of 1500. Elapsed time 8.51 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.59 seconds. BPR loss 4.12E-02. Sample per second: 15179\n",
      "MF_BPR: Epoch 41 of 1500. Elapsed time 8.59 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.87 seconds. BPR loss 4.07E-02. Sample per second: 14298\n",
      "MF_BPR: Epoch 42 of 1500. Elapsed time 8.66 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.15 seconds. BPR loss 4.10E-02. Sample per second: 13527\n",
      "MF_BPR: Epoch 43 of 1500. Elapsed time 8.73 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.48 seconds. BPR loss 4.12E-02. Sample per second: 15559\n",
      "MF_BPR: Epoch 44 of 1500. Elapsed time 8.80 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.74 seconds. BPR loss 4.16E-02. Sample per second: 14679\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.56 sec. Users per second: 1669\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.77 sec. Users per second: 1667\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0126385, PRECISION: 0.0025705, PRECISION_RECALL_MIN_DEN: 0.0028299, RECALL: 0.0011574, MAP: 0.0008900, MRR: 0.0077859, NDCG: 0.0011209, F1: 0.0015962, HIT_RATE: 0.0257055, ARHR: 0.0079426, NOVELTY: 0.0149121, AVERAGE_POPULARITY: 0.0276585, DIVERSITY_MEAN_INTER_LIST: 0.9987864, DIVERSITY_HERFINDAHL: 0.9998772, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0966202, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0246286, DIVERSITY_GINI: 0.7007393, SHANNON_ENTROPY: 13.1738317, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 45 of 1500. Elapsed time 9.57 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.87 seconds. BPR loss 4.15E-02. Sample per second: 14302\n",
      "MF_BPR: Epoch 46 of 1500. Elapsed time 9.64 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.13 seconds. BPR loss 4.16E-02. Sample per second: 13577\n",
      "MF_BPR: Epoch 47 of 1500. Elapsed time 9.71 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.41 seconds. BPR loss 4.17E-02. Sample per second: 15780\n",
      "MF_BPR: Epoch 48 of 1500. Elapsed time 9.78 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.80 seconds. BPR loss 4.16E-02. Sample per second: 14521\n",
      "MF_BPR: Epoch 49 of 1500. Elapsed time 9.86 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.23 seconds. BPR loss 4.15E-02. Sample per second: 13306\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.17 sec. Users per second: 1624\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.66 sec. Users per second: 1632\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0131547, PRECISION: 0.0026869, PRECISION_RECALL_MIN_DEN: 0.0029630, RECALL: 0.0012264, MAP: 0.0009362, MRR: 0.0081335, NDCG: 0.0012020, F1: 0.0016841, HIT_RATE: 0.0268687, ARHR: 0.0083072, NOVELTY: 0.0148946, AVERAGE_POPULARITY: 0.0286375, DIVERSITY_MEAN_INTER_LIST: 0.9987848, DIVERSITY_HERFINDAHL: 0.9998770, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0973692, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0256733, DIVERSITY_GINI: 0.7001125, SHANNON_ENTROPY: 13.1728853, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 50 of 1500. Elapsed time 10.64 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.35 seconds. BPR loss 4.24E-02. Sample per second: 15995\n",
      "MF_BPR: Epoch 51 of 1500. Elapsed time 10.72 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.80 seconds. BPR loss 4.24E-02. Sample per second: 14519\n",
      "MF_BPR: Epoch 52 of 1500. Elapsed time 10.79 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.18 seconds. BPR loss 4.20E-02. Sample per second: 13431\n",
      "MF_BPR: Epoch 53 of 1500. Elapsed time 10.86 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.52 seconds. BPR loss 4.18E-02. Sample per second: 15401\n",
      "MF_BPR: Epoch 54 of 1500. Elapsed time 10.93 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.91 seconds. BPR loss 4.21E-02. Sample per second: 14184\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 47000 ( 67.49% ) in 30.19 sec. Users per second: 1557\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 44.31 sec. Users per second: 1572\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0139238, PRECISION: 0.0028391, PRECISION_RECALL_MIN_DEN: 0.0031507, RECALL: 0.0013438, MAP: 0.0009958, MRR: 0.0086228, NDCG: 0.0013148, F1: 0.0018241, HIT_RATE: 0.0283909, ARHR: 0.0088091, NOVELTY: 0.0148715, AVERAGE_POPULARITY: 0.0299267, DIVERSITY_MEAN_INTER_LIST: 0.9987827, DIVERSITY_HERFINDAHL: 0.9998768, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0979309, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0271473, DIVERSITY_GINI: 0.6995372, SHANNON_ENTROPY: 13.1718992, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 55 of 1500. Elapsed time 11.75 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.72 seconds. BPR loss 4.25E-02. Sample per second: 14746\n",
      "MF_BPR: Epoch 56 of 1500. Elapsed time 11.82 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.15 seconds. BPR loss 4.24E-02. Sample per second: 13520\n",
      "MF_BPR: Epoch 57 of 1500. Elapsed time 11.90 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.61 seconds. BPR loss 4.29E-02. Sample per second: 15095\n",
      "MF_BPR: Epoch 58 of 1500. Elapsed time 11.97 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.03 seconds. BPR loss 4.27E-02. Sample per second: 13838\n",
      "MF_BPR: Epoch 59 of 1500. Elapsed time 12.04 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.40 seconds. BPR loss 4.28E-02. Sample per second: 15838\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 46000 ( 66.06% ) in 30.05 sec. Users per second: 1531\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 44.12 sec. Users per second: 1578\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0149011, PRECISION: 0.0030129, PRECISION_RECALL_MIN_DEN: 0.0033615, RECALL: 0.0014789, MAP: 0.0010700, MRR: 0.0091889, NDCG: 0.0014445, F1: 0.0019840, HIT_RATE: 0.0301285, ARHR: 0.0093846, NOVELTY: 0.0148457, AVERAGE_POPULARITY: 0.0314804, DIVERSITY_MEAN_INTER_LIST: 0.9987785, DIVERSITY_HERFINDAHL: 0.9998764, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0987735, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0288360, DIVERSITY_GINI: 0.6984791, SHANNON_ENTROPY: 13.1700526, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 60 of 1500. Elapsed time 12.85 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.97 seconds. BPR loss 4.34E-02. Sample per second: 14000\n",
      "MF_BPR: Epoch 61 of 1500. Elapsed time 12.93 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.33 seconds. BPR loss 4.33E-02. Sample per second: 13076\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 62 of 1500. Elapsed time 13.00 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.66 seconds. BPR loss 4.37E-02. Sample per second: 14946\n",
      "MF_BPR: Epoch 63 of 1500. Elapsed time 13.07 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.93 seconds. BPR loss 4.38E-02. Sample per second: 14118\n",
      "MF_BPR: Epoch 64 of 1500. Elapsed time 13.14 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.18 seconds. BPR loss 4.38E-02. Sample per second: 13437\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.28 sec. Users per second: 1651\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.02 sec. Users per second: 1657\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0161711, PRECISION: 0.0032469, PRECISION_RECALL_MIN_DEN: 0.0036670, RECALL: 0.0016970, MAP: 0.0011880, MRR: 0.0100227, NDCG: 0.0016524, F1: 0.0022290, HIT_RATE: 0.0324693, ARHR: 0.0102648, NOVELTY: 0.0148126, AVERAGE_POPULARITY: 0.0335902, DIVERSITY_MEAN_INTER_LIST: 0.9987694, DIVERSITY_HERFINDAHL: 0.9998755, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0994289, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0308681, DIVERSITY_GINI: 0.6963221, SHANNON_ENTROPY: 13.1661340, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 65 of 1500. Elapsed time 13.91 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.69 seconds. BPR loss 4.38E-02. Sample per second: 14845\n",
      "MF_BPR: Epoch 66 of 1500. Elapsed time 13.99 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.97 seconds. BPR loss 4.41E-02. Sample per second: 14014\n",
      "MF_BPR: Epoch 67 of 1500. Elapsed time 14.06 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.31 seconds. BPR loss 4.41E-02. Sample per second: 13102\n",
      "MF_BPR: Epoch 68 of 1500. Elapsed time 14.13 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.65 seconds. BPR loss 4.45E-02. Sample per second: 14971\n",
      "MF_BPR: Epoch 69 of 1500. Elapsed time 14.20 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.95 seconds. BPR loss 4.48E-02. Sample per second: 14055\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.26 sec. Users per second: 1620\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.57 sec. Users per second: 1636\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0179245, PRECISION: 0.0035485, PRECISION_RECALL_MIN_DEN: 0.0040734, RECALL: 0.0019883, MAP: 0.0013504, MRR: 0.0111783, NDCG: 0.0019270, F1: 0.0025486, HIT_RATE: 0.0354850, ARHR: 0.0114554, NOVELTY: 0.0147720, AVERAGE_POPULARITY: 0.0362662, DIVERSITY_MEAN_INTER_LIST: 0.9987540, DIVERSITY_HERFINDAHL: 0.9998740, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0996161, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0336873, DIVERSITY_GINI: 0.6934248, SHANNON_ENTROPY: 13.1602942, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 70 of 1500. Elapsed time 14.99 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.01 seconds. BPR loss 4.49E-02. Sample per second: 13896\n",
      "MF_BPR: Epoch 71 of 1500. Elapsed time 15.06 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.45 seconds. BPR loss 4.53E-02. Sample per second: 15655\n",
      "MF_BPR: Epoch 72 of 1500. Elapsed time 15.13 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.89 seconds. BPR loss 4.51E-02. Sample per second: 14231\n",
      "MF_BPR: Epoch 73 of 1500. Elapsed time 15.21 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.33 seconds. BPR loss 4.59E-02. Sample per second: 13060\n",
      "MF_BPR: Epoch 74 of 1500. Elapsed time 15.28 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.78 seconds. BPR loss 4.63E-02. Sample per second: 14577\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.93% ) in 30.20 sec. Users per second: 1589\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.50 sec. Users per second: 1601\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0204030, PRECISION: 0.0039894, PRECISION_RECALL_MIN_DEN: 0.0046533, RECALL: 0.0024057, MAP: 0.0015942, MRR: 0.0128371, NDCG: 0.0023431, F1: 0.0030014, HIT_RATE: 0.0398937, ARHR: 0.0131825, NOVELTY: 0.0147237, AVERAGE_POPULARITY: 0.0397214, DIVERSITY_MEAN_INTER_LIST: 0.9987251, DIVERSITY_HERFINDAHL: 0.9998711, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0999906, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0377229, DIVERSITY_GINI: 0.6894072, SHANNON_ENTROPY: 13.1508142, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 75 of 1500. Elapsed time 16.08 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.69 seconds. BPR loss 4.62E-02. Sample per second: 14858\n",
      "MF_BPR: Epoch 76 of 1500. Elapsed time 16.15 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.01 seconds. BPR loss 4.64E-02. Sample per second: 13890\n",
      "MF_BPR: Epoch 77 of 1500. Elapsed time 16.23 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.36 seconds. BPR loss 4.63E-02. Sample per second: 15985\n",
      "MF_BPR: Epoch 78 of 1500. Elapsed time 16.30 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.70 seconds. BPR loss 4.67E-02. Sample per second: 14814\n",
      "MF_BPR: Epoch 79 of 1500. Elapsed time 16.37 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.14 seconds. BPR loss 4.76E-02. Sample per second: 13544\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.93% ) in 30.15 sec. Users per second: 1592\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 45.36 sec. Users per second: 1535\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0237528, PRECISION: 0.0045911, PRECISION_RECALL_MIN_DEN: 0.0054341, RECALL: 0.0029720, MAP: 0.0019461, MRR: 0.0150917, NDCG: 0.0029158, F1: 0.0036083, HIT_RATE: 0.0459108, ARHR: 0.0155730, NOVELTY: 0.0146638, AVERAGE_POPULARITY: 0.0441107, DIVERSITY_MEAN_INTER_LIST: 0.9986733, DIVERSITY_HERFINDAHL: 0.9998659, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.1008333, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0430035, DIVERSITY_GINI: 0.6839879, SHANNON_ENTROPY: 13.1362543, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 80 of 1500. Elapsed time 17.20 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.22 seconds. BPR loss 4.81E-02. Sample per second: 13340\n",
      "MF_BPR: Epoch 81 of 1500. Elapsed time 17.28 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.71 seconds. BPR loss 4.76E-02. Sample per second: 14787\n",
      "MF_BPR: Epoch 82 of 1500. Elapsed time 17.35 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.43 seconds. BPR loss 4.84E-02. Sample per second: 12832\n",
      "MF_BPR: Epoch 83 of 1500. Elapsed time 17.43 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.35 seconds. BPR loss 4.91E-02. Sample per second: 13020\n",
      "MF_BPR: Epoch 84 of 1500. Elapsed time 17.52 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.03 seconds. BPR loss 4.88E-02. Sample per second: 13837\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.33 sec. Users per second: 1616\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.42 sec. Users per second: 1604\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0279238, PRECISION: 0.0053350, PRECISION_RECALL_MIN_DEN: 0.0064333, RECALL: 0.0037241, MAP: 0.0024229, MRR: 0.0179946, NDCG: 0.0036905, F1: 0.0043863, HIT_RATE: 0.0533496, ARHR: 0.0187398, NOVELTY: 0.0145891, AVERAGE_POPULARITY: 0.0497863, DIVERSITY_MEAN_INTER_LIST: 0.9985830, DIVERSITY_HERFINDAHL: 0.9998569, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.1000843, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0490999, DIVERSITY_GINI: 0.6768738, SHANNON_ENTROPY: 13.1143033, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 85 of 1500. Elapsed time 18.32 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.96 seconds. BPR loss 4.96E-02. Sample per second: 14044\n",
      "MF_BPR: Epoch 86 of 1500. Elapsed time 18.39 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.32 seconds. BPR loss 4.99E-02. Sample per second: 13086\n",
      "MF_BPR: Epoch 87 of 1500. Elapsed time 18.46 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.72 seconds. BPR loss 5.06E-02. Sample per second: 14761\n",
      "MF_BPR: Epoch 88 of 1500. Elapsed time 18.54 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.35 seconds. BPR loss 5.08E-02. Sample per second: 13027\n",
      "MF_BPR: Epoch 89 of 1500. Elapsed time 18.62 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.75 seconds. BPR loss 5.19E-02. Sample per second: 14673\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.47 sec. Users per second: 1641\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.41 sec. Users per second: 1642\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0329388, PRECISION: 0.0063890, PRECISION_RECALL_MIN_DEN: 0.0079898, RECALL: 0.0049640, MAP: 0.0030952, MRR: 0.0218747, NDCG: 0.0048003, F1: 0.0055871, HIT_RATE: 0.0638903, ARHR: 0.0230852, NOVELTY: 0.0144973, AVERAGE_POPULARITY: 0.0569935, DIVERSITY_MEAN_INTER_LIST: 0.9984261, DIVERSITY_HERFINDAHL: 0.9998412, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0998034, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0571854, DIVERSITY_GINI: 0.6679294, SHANNON_ENTROPY: 13.0823090, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 90 of 1500. Elapsed time 19.40 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.69 seconds. BPR loss 5.25E-02. Sample per second: 14839\n",
      "MF_BPR: Epoch 91 of 1500. Elapsed time 19.47 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.02 seconds. BPR loss 5.28E-02. Sample per second: 13863\n",
      "MF_BPR: Epoch 92 of 1500. Elapsed time 19.54 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.52 seconds. BPR loss 5.31E-02. Sample per second: 15400\n",
      "MF_BPR: Epoch 93 of 1500. Elapsed time 19.62 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.02 seconds. BPR loss 5.40E-02. Sample per second: 13873\n",
      "MF_BPR: Epoch 94 of 1500. Elapsed time 19.69 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.43 seconds. BPR loss 5.40E-02. Sample per second: 15731\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.10 sec. Users per second: 1628\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.52 sec. Users per second: 1638\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0396633, PRECISION: 0.0077806, PRECISION_RECALL_MIN_DEN: 0.0100640, RECALL: 0.0066305, MAP: 0.0040186, MRR: 0.0268631, NDCG: 0.0062951, F1: 0.0071597, HIT_RATE: 0.0778057, ARHR: 0.0287966, NOVELTY: 0.0143833, AVERAGE_POPULARITY: 0.0662269, DIVERSITY_MEAN_INTER_LIST: 0.9981460, DIVERSITY_HERFINDAHL: 0.9998132, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.1006460, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0677466, DIVERSITY_GINI: 0.6563716, SHANNON_ENTROPY: 13.0350956, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 95 of 1500. Elapsed time 20.48 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.42 seconds. BPR loss 5.51E-02. Sample per second: 15769\n",
      "MF_BPR: Epoch 96 of 1500. Elapsed time 20.55 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.82 seconds. BPR loss 5.53E-02. Sample per second: 14458\n",
      "MF_BPR: Epoch 97 of 1500. Elapsed time 20.62 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.14 seconds. BPR loss 5.68E-02. Sample per second: 13552\n",
      "MF_BPR: Epoch 98 of 1500. Elapsed time 20.69 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.44 seconds. BPR loss 5.70E-02. Sample per second: 15697\n",
      "MF_BPR: Epoch 99 of 1500. Elapsed time 20.77 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.71 seconds. BPR loss 5.77E-02. Sample per second: 14792\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.05 sec. Users per second: 1664\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.86 sec. Users per second: 1664\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0477946, PRECISION: 0.0095297, PRECISION_RECALL_MIN_DEN: 0.0127203, RECALL: 0.0088050, MAP: 0.0052509, MRR: 0.0328749, NDCG: 0.0082512, F1: 0.0091530, HIT_RATE: 0.0952969, ARHR: 0.0359172, NOVELTY: 0.0142497, AVERAGE_POPULARITY: 0.0773404, DIVERSITY_MEAN_INTER_LIST: 0.9977003, DIVERSITY_HERFINDAHL: 0.9997686, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.1006460, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0800538, DIVERSITY_GINI: 0.6422386, SHANNON_ENTROPY: 12.9710041, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 100 of 1500. Elapsed time 21.54 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.99 seconds. BPR loss 5.89E-02. Sample per second: 13957\n",
      "MF_BPR: Epoch 101 of 1500. Elapsed time 21.61 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.41 seconds. BPR loss 5.96E-02. Sample per second: 12867\n",
      "MF_BPR: Epoch 102 of 1500. Elapsed time 21.68 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.79 seconds. BPR loss 6.04E-02. Sample per second: 14541\n",
      "MF_BPR: Epoch 103 of 1500. Elapsed time 21.76 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.12 seconds. BPR loss 6.13E-02. Sample per second: 13595\n",
      "MF_BPR: Epoch 104 of 1500. Elapsed time 21.83 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.44 seconds. BPR loss 6.20E-02. Sample per second: 15670\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.06 sec. Users per second: 1630\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.59 sec. Users per second: 1597\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0563277, PRECISION: 0.0116105, PRECISION_RECALL_MIN_DEN: 0.0158087, RECALL: 0.0113083, MAP: 0.0067417, MRR: 0.0394864, NDCG: 0.0104785, F1: 0.0114574, HIT_RATE: 0.1161054, ARHR: 0.0440369, NOVELTY: 0.0140935, AVERAGE_POPULARITY: 0.0906411, DIVERSITY_MEAN_INTER_LIST: 0.9970039, DIVERSITY_HERFINDAHL: 0.9996990, COVERAGE_ITEM: 0.9999064, COVERAGE_ITEM_CORRECT: 0.1005524, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0933627, DIVERSITY_GINI: 0.6255019, SHANNON_ENTROPY: 12.8865391, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 105 of 1500. Elapsed time 22.63 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.58 seconds. BPR loss 6.37E-02. Sample per second: 15189\n",
      "MF_BPR: Epoch 106 of 1500. Elapsed time 22.70 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.93 seconds. BPR loss 6.45E-02. Sample per second: 14130\n",
      "MF_BPR: Epoch 107 of 1500. Elapsed time 22.77 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.26 seconds. BPR loss 6.58E-02. Sample per second: 13240\n",
      "MF_BPR: Epoch 108 of 1500. Elapsed time 22.85 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.59 seconds. BPR loss 6.67E-02. Sample per second: 15177\n",
      "MF_BPR: Epoch 109 of 1500. Elapsed time 22.92 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.93 seconds. BPR loss 6.82E-02. Sample per second: 14113\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.06 sec. Users per second: 1630\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.75 sec. Users per second: 1629\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0653250, PRECISION: 0.0141524, PRECISION_RECALL_MIN_DEN: 0.0194934, RECALL: 0.0142533, MAP: 0.0084832, MRR: 0.0466948, NDCG: 0.0130862, F1: 0.0142027, HIT_RATE: 0.1415237, ARHR: 0.0533290, NOVELTY: 0.0139140, AVERAGE_POPULARITY: 0.1061525, DIVERSITY_MEAN_INTER_LIST: 0.9959524, DIVERSITY_HERFINDAHL: 0.9995938, COVERAGE_ITEM: 0.9999064, COVERAGE_ITEM_CORRECT: 0.0993353, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1080025, DIVERSITY_GINI: 0.6060461, SHANNON_ENTROPY: 12.7782458, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 110 of 1500. Elapsed time 23.71 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.11 seconds. BPR loss 6.84E-02. Sample per second: 13618\n",
      "MF_BPR: Epoch 111 of 1500. Elapsed time 23.78 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.40 seconds. BPR loss 7.07E-02. Sample per second: 15817\n",
      "MF_BPR: Epoch 112 of 1500. Elapsed time 23.85 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.72 seconds. BPR loss 7.12E-02. Sample per second: 14764\n",
      "MF_BPR: Epoch 113 of 1500. Elapsed time 23.92 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.02 seconds. BPR loss 7.36E-02. Sample per second: 13869\n",
      "MF_BPR: Epoch 114 of 1500. Elapsed time 23.99 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 7.57E-02. Sample per second: 15935\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.93% ) in 30.53 sec. Users per second: 1572\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.79 sec. Users per second: 1590\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0746020, PRECISION: 0.0169225, PRECISION_RECALL_MIN_DEN: 0.0234926, RECALL: 0.0174336, MAP: 0.0103773, MRR: 0.0542611, NDCG: 0.0158662, F1: 0.0171743, HIT_RATE: 0.1692252, ARHR: 0.0632796, NOVELTY: 0.0137147, AVERAGE_POPULARITY: 0.1234928, DIVERSITY_MEAN_INTER_LIST: 0.9944836, DIVERSITY_HERFINDAHL: 0.9994469, COVERAGE_ITEM: 0.9999064, COVERAGE_ITEM_CORRECT: 0.0986799, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1235439, DIVERSITY_GINI: 0.5845042, SHANNON_ENTROPY: 12.6482753, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 115 of 1500. Elapsed time 24.80 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.51 seconds. BPR loss 7.58E-02. Sample per second: 15437\n",
      "MF_BPR: Epoch 116 of 1500. Elapsed time 24.87 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.71 seconds. BPR loss 7.84E-02. Sample per second: 14770\n",
      "MF_BPR: Epoch 117 of 1500. Elapsed time 24.94 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.91 seconds. BPR loss 7.97E-02. Sample per second: 14172\n",
      "MF_BPR: Epoch 118 of 1500. Elapsed time 25.01 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.11 seconds. BPR loss 8.10E-02. Sample per second: 13627\n",
      "MF_BPR: Epoch 119 of 1500. Elapsed time 25.08 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 8.30E-02. Sample per second: 15951\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.93% ) in 30.05 sec. Users per second: 1598\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.44 sec. Users per second: 1641\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0843067, PRECISION: 0.0199828, PRECISION_RECALL_MIN_DEN: 0.0278187, RECALL: 0.0208053, MAP: 0.0123716, MRR: 0.0620660, NDCG: 0.0187356, F1: 0.0203857, HIT_RATE: 0.1998277, ARHR: 0.0738796, NOVELTY: 0.0135042, AVERAGE_POPULARITY: 0.1420592, DIVERSITY_MEAN_INTER_LIST: 0.9925658, DIVERSITY_HERFINDAHL: 0.9992552, COVERAGE_ITEM: 0.9999064, COVERAGE_ITEM_CORRECT: 0.0977437, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1399725, DIVERSITY_GINI: 0.5615602, SHANNON_ENTROPY: 12.4994431, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 120 of 1500. Elapsed time 25.86 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.10 seconds. BPR loss 8.46E-02. Sample per second: 13659\n",
      "MF_BPR: Epoch 121 of 1500. Elapsed time 25.93 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.32 seconds. BPR loss 8.78E-02. Sample per second: 16132\n",
      "MF_BPR: Epoch 122 of 1500. Elapsed time 26.00 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.53 seconds. BPR loss 8.89E-02. Sample per second: 15372\n",
      "MF_BPR: Epoch 123 of 1500. Elapsed time 26.07 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.79 seconds. BPR loss 9.24E-02. Sample per second: 14535\n",
      "MF_BPR: Epoch 124 of 1500. Elapsed time 26.14 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.05 seconds. BPR loss 9.49E-02. Sample per second: 13789\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.62 sec. Users per second: 1633\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.38 sec. Users per second: 1605\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0941247, PRECISION: 0.0231895, PRECISION_RECALL_MIN_DEN: 0.0322588, RECALL: 0.0242064, MAP: 0.0143845, MRR: 0.0698821, NDCG: 0.0215840, F1: 0.0236870, HIT_RATE: 0.2318949, ARHR: 0.0846073, NOVELTY: 0.0132824, AVERAGE_POPULARITY: 0.1619136, DIVERSITY_MEAN_INTER_LIST: 0.9900909, DIVERSITY_HERFINDAHL: 0.9990077, COVERAGE_ITEM: 0.9999064, COVERAGE_ITEM_CORRECT: 0.0952158, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1570594, DIVERSITY_GINI: 0.5375187, SHANNON_ENTROPY: 12.3311653, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 125 of 1500. Elapsed time 26.94 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.85 seconds. BPR loss 9.82E-02. Sample per second: 14353\n",
      "MF_BPR: Epoch 126 of 1500. Elapsed time 27.01 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.05 seconds. BPR loss 9.85E-02. Sample per second: 13789\n",
      "MF_BPR: Epoch 127 of 1500. Elapsed time 27.08 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.30 seconds. BPR loss 1.02E-01. Sample per second: 16179\n",
      "MF_BPR: Epoch 128 of 1500. Elapsed time 27.15 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.53 seconds. BPR loss 1.04E-01. Sample per second: 15370\n",
      "MF_BPR: Epoch 129 of 1500. Elapsed time 27.22 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.76 seconds. BPR loss 1.08E-01. Sample per second: 14640\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 47000 ( 67.49% ) in 30.40 sec. Users per second: 1546\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.92 sec. Users per second: 1586\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1035502, PRECISION: 0.0266073, PRECISION_RECALL_MIN_DEN: 0.0368122, RECALL: 0.0275714, MAP: 0.0164286, MRR: 0.0776212, NDCG: 0.0244895, F1: 0.0270808, HIT_RATE: 0.2660731, ARHR: 0.0955775, NOVELTY: 0.0130591, AVERAGE_POPULARITY: 0.1819888, DIVERSITY_MEAN_INTER_LIST: 0.9871453, DIVERSITY_HERFINDAHL: 0.9987131, COVERAGE_ITEM: 0.9999064, COVERAGE_ITEM_CORRECT: 0.0948413, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1735310, DIVERSITY_GINI: 0.5130015, SHANNON_ENTROPY: 12.1508836, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 130 of 1500. Elapsed time 28.02 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.17 seconds. BPR loss 1.12E-01. Sample per second: 13463\n",
      "MF_BPR: Epoch 131 of 1500. Elapsed time 28.10 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.62 seconds. BPR loss 1.14E-01. Sample per second: 15087\n",
      "MF_BPR: Epoch 132 of 1500. Elapsed time 28.17 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.10 seconds. BPR loss 1.15E-01. Sample per second: 13666\n",
      "MF_BPR: Epoch 133 of 1500. Elapsed time 28.24 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.45 seconds. BPR loss 1.19E-01. Sample per second: 15656\n",
      "MF_BPR: Epoch 134 of 1500. Elapsed time 28.32 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.67 seconds. BPR loss 1.24E-01. Sample per second: 14897\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.21 sec. Users per second: 1688\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.94 sec. Users per second: 1701\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1125563, PRECISION: 0.0299275, PRECISION_RECALL_MIN_DEN: 0.0412908, RECALL: 0.0308551, MAP: 0.0184760, MRR: 0.0852276, NDCG: 0.0272926, F1: 0.0303842, HIT_RATE: 0.2992748, ARHR: 0.1064345, NOVELTY: 0.0128374, AVERAGE_POPULARITY: 0.2020401, DIVERSITY_MEAN_INTER_LIST: 0.9837743, DIVERSITY_HERFINDAHL: 0.9983760, COVERAGE_ITEM: 0.9999064, COVERAGE_ITEM_CORRECT: 0.0936242, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1895876, DIVERSITY_GINI: 0.4889071, SHANNON_ENTROPY: 11.9638741, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 135 of 1500. Elapsed time 29.07 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.19 seconds. BPR loss 1.27E-01. Sample per second: 13428\n",
      "MF_BPR: Epoch 136 of 1500. Elapsed time 29.15 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.72 seconds. BPR loss 1.30E-01. Sample per second: 14762\n",
      "MF_BPR: Epoch 137 of 1500. Elapsed time 29.22 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.08 seconds. BPR loss 1.34E-01. Sample per second: 13716\n",
      "MF_BPR: Epoch 138 of 1500. Elapsed time 29.29 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.36 seconds. BPR loss 1.38E-01. Sample per second: 15961\n",
      "MF_BPR: Epoch 139 of 1500. Elapsed time 29.37 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.44 seconds. BPR loss 1.42E-01. Sample per second: 15684\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.38 sec. Users per second: 1646\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.86 sec. Users per second: 1663\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1206184, PRECISION: 0.0331859, PRECISION_RECALL_MIN_DEN: 0.0455277, RECALL: 0.0338799, MAP: 0.0204768, MRR: 0.0924329, NDCG: 0.0299512, F1: 0.0335293, HIT_RATE: 0.3318590, ARHR: 0.1170273, NOVELTY: 0.0126188, AVERAGE_POPULARITY: 0.2218607, DIVERSITY_MEAN_INTER_LIST: 0.9799998, DIVERSITY_HERFINDAHL: 0.9979986, COVERAGE_ITEM: 0.9999064, COVERAGE_ITEM_CORRECT: 0.0925007, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2041272, DIVERSITY_GINI: 0.4648880, SHANNON_ENTROPY: 11.7706076, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 140 of 1500. Elapsed time 30.13 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.86 seconds. BPR loss 1.49E-01. Sample per second: 14342\n",
      "MF_BPR: Epoch 141 of 1500. Elapsed time 30.21 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.37 seconds. BPR loss 1.52E-01. Sample per second: 12971\n",
      "MF_BPR: Epoch 142 of 1500. Elapsed time 30.28 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.73 seconds. BPR loss 1.56E-01. Sample per second: 14713\n",
      "MF_BPR: Epoch 143 of 1500. Elapsed time 30.35 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.95 seconds. BPR loss 1.61E-01. Sample per second: 14065\n",
      "MF_BPR: Epoch 144 of 1500. Elapsed time 30.43 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.03 seconds. BPR loss 1.67E-01. Sample per second: 13853\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.38 sec. Users per second: 1744\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.72 sec. Users per second: 1753\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1279662, PRECISION: 0.0364371, PRECISION_RECALL_MIN_DEN: 0.0497414, RECALL: 0.0368223, MAP: 0.0223389, MRR: 0.0991529, NDCG: 0.0324752, F1: 0.0366287, HIT_RATE: 0.3643714, ARHR: 0.1269955, NOVELTY: 0.0124092, AVERAGE_POPULARITY: 0.2411750, DIVERSITY_MEAN_INTER_LIST: 0.9758341, DIVERSITY_HERFINDAHL: 0.9975820, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0922198, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2186239, DIVERSITY_GINI: 0.4417236, SHANNON_ENTROPY: 11.5756868, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 145 of 1500. Elapsed time 31.16 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.97 seconds. BPR loss 1.69E-01. Sample per second: 14014\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 146 of 1500. Elapsed time 31.23 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.11 seconds. BPR loss 1.76E-01. Sample per second: 13628\n",
      "MF_BPR: Epoch 147 of 1500. Elapsed time 31.29 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.38 seconds. BPR loss 1.83E-01. Sample per second: 15897\n",
      "MF_BPR: Epoch 148 of 1500. Elapsed time 31.37 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.79 seconds. BPR loss 1.86E-01. Sample per second: 14543\n",
      "MF_BPR: Epoch 149 of 1500. Elapsed time 31.44 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.02 seconds. BPR loss 1.93E-01. Sample per second: 13862\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.01 sec. Users per second: 1633\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.20 sec. Users per second: 1690\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1350449, PRECISION: 0.0395376, PRECISION_RECALL_MIN_DEN: 0.0536630, RECALL: 0.0394879, MAP: 0.0241316, MRR: 0.1055567, NDCG: 0.0348141, F1: 0.0395127, HIT_RATE: 0.3953759, ARHR: 0.1365784, NOVELTY: 0.0122071, AVERAGE_POPULARITY: 0.2597700, DIVERSITY_MEAN_INTER_LIST: 0.9714842, DIVERSITY_HERFINDAHL: 0.9971470, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0909091, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2326197, DIVERSITY_GINI: 0.4194948, SHANNON_ENTROPY: 11.3832254, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 150 of 1500. Elapsed time 32.20 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.33 seconds. BPR loss 1.99E-01. Sample per second: 16079\n",
      "MF_BPR: Epoch 151 of 1500. Elapsed time 32.26 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.34 seconds. BPR loss 2.04E-01. Sample per second: 16030\n",
      "MF_BPR: Epoch 152 of 1500. Elapsed time 32.33 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.45 seconds. BPR loss 2.13E-01. Sample per second: 15645\n",
      "MF_BPR: Epoch 153 of 1500. Elapsed time 32.40 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.62 seconds. BPR loss 2.18E-01. Sample per second: 15063\n",
      "MF_BPR: Epoch 154 of 1500. Elapsed time 32.47 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.96 seconds. BPR loss 2.24E-01. Sample per second: 14030\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.19 sec. Users per second: 1656\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.81 sec. Users per second: 1666\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1417180, PRECISION: 0.0422819, PRECISION_RECALL_MIN_DEN: 0.0572322, RECALL: 0.0419090, MAP: 0.0258214, MRR: 0.1113160, NDCG: 0.0370073, F1: 0.0420946, HIT_RATE: 0.4228190, ARHR: 0.1452841, NOVELTY: 0.0120190, AVERAGE_POPULARITY: 0.2772690, DIVERSITY_MEAN_INTER_LIST: 0.9670300, DIVERSITY_HERFINDAHL: 0.9967016, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0894111, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2444260, DIVERSITY_GINI: 0.3987705, SHANNON_ENTROPY: 11.1975617, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 155 of 1500. Elapsed time 33.24 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.49 seconds. BPR loss 2.31E-01. Sample per second: 12672\n",
      "MF_BPR: Epoch 156 of 1500. Elapsed time 33.32 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.00 seconds. BPR loss 2.37E-01. Sample per second: 13923\n",
      "MF_BPR: Epoch 157 of 1500. Elapsed time 33.39 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.08 seconds. BPR loss 2.48E-01. Sample per second: 17049\n",
      "MF_BPR: Epoch 158 of 1500. Elapsed time 33.46 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.15 seconds. BPR loss 2.53E-01. Sample per second: 16795\n",
      "MF_BPR: Epoch 159 of 1500. Elapsed time 33.53 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.31 seconds. BPR loss 2.66E-01. Sample per second: 16143\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.42 sec. Users per second: 1643\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.38 sec. Users per second: 1605\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1471321, PRECISION: 0.0450147, PRECISION_RECALL_MIN_DEN: 0.0605810, RECALL: 0.0440792, MAP: 0.0273425, MRR: 0.1165673, NDCG: 0.0389874, F1: 0.0445420, HIT_RATE: 0.4501472, ARHR: 0.1534277, NOVELTY: 0.0118377, AVERAGE_POPULARITY: 0.2943204, DIVERSITY_MEAN_INTER_LIST: 0.9624301, DIVERSITY_HERFINDAHL: 0.9962416, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0879131, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2563038, DIVERSITY_GINI: 0.3788872, SHANNON_ENTROPY: 11.0146771, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 160 of 1500. Elapsed time 34.32 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.20 seconds. BPR loss 2.76E-01. Sample per second: 13403\n",
      "MF_BPR: Epoch 161 of 1500. Elapsed time 34.40 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.52 seconds. BPR loss 2.84E-01. Sample per second: 15392\n",
      "MF_BPR: Epoch 162 of 1500. Elapsed time 34.47 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.95 seconds. BPR loss 2.85E-01. Sample per second: 14060\n",
      "MF_BPR: Epoch 163 of 1500. Elapsed time 34.54 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.36 seconds. BPR loss 3.01E-01. Sample per second: 13003\n",
      "MF_BPR: Epoch 164 of 1500. Elapsed time 34.62 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.71 seconds. BPR loss 3.11E-01. Sample per second: 14784\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.35 sec. Users per second: 1614\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.57 sec. Users per second: 1598\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1527818, PRECISION: 0.0475307, PRECISION_RECALL_MIN_DEN: 0.0637090, RECALL: 0.0460862, MAP: 0.0287998, MRR: 0.1216927, NDCG: 0.0408406, F1: 0.0467973, HIT_RATE: 0.4753070, ARHR: 0.1611456, NOVELTY: 0.0116711, AVERAGE_POPULARITY: 0.3101840, DIVERSITY_MEAN_INTER_LIST: 0.9578353, DIVERSITY_HERFINDAHL: 0.9957822, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0860406, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2673230, DIVERSITY_GINI: 0.3602994, SHANNON_ENTROPY: 10.8402096, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 165 of 1500. Elapsed time 35.42 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.60 seconds. BPR loss 3.15E-01. Sample per second: 15130\n",
      "MF_BPR: Epoch 166 of 1500. Elapsed time 35.49 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.77 seconds. BPR loss 3.25E-01. Sample per second: 14599\n",
      "MF_BPR: Epoch 167 of 1500. Elapsed time 35.56 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.99 seconds. BPR loss 3.38E-01. Sample per second: 13951\n",
      "MF_BPR: Epoch 168 of 1500. Elapsed time 35.63 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.27 seconds. BPR loss 3.51E-01. Sample per second: 13205\n",
      "MF_BPR: Epoch 169 of 1500. Elapsed time 35.70 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.43 seconds. BPR loss 3.62E-01. Sample per second: 15721\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.27 sec. Users per second: 1685\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.96 sec. Users per second: 1700\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1581687, PRECISION: 0.0499203, PRECISION_RECALL_MIN_DEN: 0.0665707, RECALL: 0.0478250, MAP: 0.0301146, MRR: 0.1265146, NDCG: 0.0424942, F1: 0.0488502, HIT_RATE: 0.4992030, ARHR: 0.1683137, NOVELTY: 0.0115152, AVERAGE_POPULARITY: 0.3251336, DIVERSITY_MEAN_INTER_LIST: 0.9532401, DIVERSITY_HERFINDAHL: 0.9953226, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0851044, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2779416, DIVERSITY_GINI: 0.3431194, SHANNON_ENTROPY: 10.6743333, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 170 of 1500. Elapsed time 36.45 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.57 seconds. BPR loss 3.73E-01. Sample per second: 15244\n",
      "MF_BPR: Epoch 171 of 1500. Elapsed time 36.52 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.68 seconds. BPR loss 3.80E-01. Sample per second: 14865\n",
      "MF_BPR: Epoch 172 of 1500. Elapsed time 36.59 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.79 seconds. BPR loss 3.88E-01. Sample per second: 14542\n",
      "MF_BPR: Epoch 173 of 1500. Elapsed time 36.66 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.95 seconds. BPR loss 3.96E-01. Sample per second: 14057\n",
      "MF_BPR: Epoch 174 of 1500. Elapsed time 36.73 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.09 seconds. BPR loss 4.15E-01. Sample per second: 13672\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.09 sec. Users per second: 1628\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.49 sec. Users per second: 1639\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1632959, PRECISION: 0.0520701, PRECISION_RECALL_MIN_DEN: 0.0691769, RECALL: 0.0493793, MAP: 0.0313402, MRR: 0.1308210, NDCG: 0.0439390, F1: 0.0506890, HIT_RATE: 0.5207008, ARHR: 0.1747681, NOVELTY: 0.0113681, AVERAGE_POPULARITY: 0.3392201, DIVERSITY_MEAN_INTER_LIST: 0.9487767, DIVERSITY_HERFINDAHL: 0.9948763, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0829510, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2873866, DIVERSITY_GINI: 0.3271218, SHANNON_ENTROPY: 10.5152160, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 175 of 1500. Elapsed time 37.50 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.86 seconds. BPR loss 4.30E-01. Sample per second: 14341\n",
      "MF_BPR: Epoch 176 of 1500. Elapsed time 37.57 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.02 seconds. BPR loss 4.39E-01. Sample per second: 13868\n",
      "MF_BPR: Epoch 177 of 1500. Elapsed time 37.64 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.56 seconds. BPR loss 4.49E-01. Sample per second: 15280\n",
      "MF_BPR: Epoch 178 of 1500. Elapsed time 37.72 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.83 seconds. BPR loss 4.62E-01. Sample per second: 14432\n",
      "MF_BPR: Epoch 179 of 1500. Elapsed time 37.79 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.97 seconds. BPR loss 4.75E-01. Sample per second: 14004\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.45 sec. Users per second: 1806\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.34 sec. Users per second: 1816\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1680330, PRECISION: 0.0540274, PRECISION_RECALL_MIN_DEN: 0.0714849, RECALL: 0.0507279, MAP: 0.0323968, MRR: 0.1347407, NDCG: 0.0451936, F1: 0.0523257, HIT_RATE: 0.5402743, ARHR: 0.1806867, NOVELTY: 0.0112329, AVERAGE_POPULARITY: 0.3523404, DIVERSITY_MEAN_INTER_LIST: 0.9443763, DIVERSITY_HERFINDAHL: 0.9944363, COVERAGE_ITEM: 0.9997191, COVERAGE_ITEM_CORRECT: 0.0814530, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2964023, DIVERSITY_GINI: 0.3123206, SHANNON_ENTROPY: 10.3648816, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 180 of 1500. Elapsed time 38.50 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.43 seconds. BPR loss 4.82E-01. Sample per second: 15702\n",
      "MF_BPR: Epoch 181 of 1500. Elapsed time 38.57 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.65 seconds. BPR loss 5.02E-01. Sample per second: 14961\n",
      "MF_BPR: Epoch 182 of 1500. Elapsed time 38.64 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.74 seconds. BPR loss 5.26E-01. Sample per second: 14696\n",
      "MF_BPR: Epoch 183 of 1500. Elapsed time 38.70 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.81 seconds. BPR loss 5.22E-01. Sample per second: 14486\n",
      "MF_BPR: Epoch 184 of 1500. Elapsed time 38.77 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.83 seconds. BPR loss 5.50E-01. Sample per second: 14429\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.32 sec. Users per second: 1748\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.12 sec. Users per second: 1780\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1725703, PRECISION: 0.0557550, PRECISION_RECALL_MIN_DEN: 0.0735238, RECALL: 0.0518991, MAP: 0.0334016, MRR: 0.1384483, NDCG: 0.0464140, F1: 0.0537580, HIT_RATE: 0.5575501, ARHR: 0.1861972, NOVELTY: 0.0111101, AVERAGE_POPULARITY: 0.3645681, DIVERSITY_MEAN_INTER_LIST: 0.9401754, DIVERSITY_HERFINDAHL: 0.9940162, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0793933, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3040728, DIVERSITY_GINI: 0.2989478, SHANNON_ENTROPY: 10.2260405, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 185 of 1500. Elapsed time 39.49 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.05 seconds. BPR loss 5.55E-01. Sample per second: 17173\n",
      "MF_BPR: Epoch 186 of 1500. Elapsed time 39.56 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.09 seconds. BPR loss 5.70E-01. Sample per second: 17043\n",
      "MF_BPR: Epoch 187 of 1500. Elapsed time 39.63 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.14 seconds. BPR loss 5.88E-01. Sample per second: 16813\n",
      "MF_BPR: Epoch 188 of 1500. Elapsed time 39.70 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.25 seconds. BPR loss 6.07E-01. Sample per second: 16388\n",
      "MF_BPR: Epoch 189 of 1500. Elapsed time 39.76 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.48 seconds. BPR loss 6.22E-01. Sample per second: 15533\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.81 sec. Users per second: 1753\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.37 sec. Users per second: 1725\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1766198, PRECISION: 0.0574050, PRECISION_RECALL_MIN_DEN: 0.0754154, RECALL: 0.0529173, MAP: 0.0343497, MRR: 0.1421134, NDCG: 0.0475687, F1: 0.0550699, HIT_RATE: 0.5740504, ARHR: 0.1916343, NOVELTY: 0.0109949, AVERAGE_POPULARITY: 0.3761053, DIVERSITY_MEAN_INTER_LIST: 0.9361415, DIVERSITY_HERFINDAHL: 0.9936128, COVERAGE_ITEM: 0.9997191, COVERAGE_ITEM_CORRECT: 0.0778953, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3111137, DIVERSITY_GINI: 0.2864177, SHANNON_ENTROPY: 10.0949438, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 190 of 1500. Elapsed time 40.51 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.36 seconds. BPR loss 6.42E-01. Sample per second: 13001\n",
      "MF_BPR: Epoch 191 of 1500. Elapsed time 40.58 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 6.50E-01. Sample per second: 15924\n",
      "MF_BPR: Epoch 192 of 1500. Elapsed time 40.65 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.49 seconds. BPR loss 6.70E-01. Sample per second: 15519\n",
      "MF_BPR: Epoch 193 of 1500. Elapsed time 40.72 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.60 seconds. BPR loss 6.91E-01. Sample per second: 15152\n",
      "MF_BPR: Epoch 194 of 1500. Elapsed time 40.79 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.81 seconds. BPR loss 7.08E-01. Sample per second: 14467\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.23 sec. Users per second: 1621\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.37 sec. Users per second: 1644\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1801635, PRECISION: 0.0589574, PRECISION_RECALL_MIN_DEN: 0.0772301, RECALL: 0.0539413, MAP: 0.0352145, MRR: 0.1453148, NDCG: 0.0486630, F1: 0.0563379, HIT_RATE: 0.5895742, ARHR: 0.1964581, NOVELTY: 0.0108895, AVERAGE_POPULARITY: 0.3867667, DIVERSITY_MEAN_INTER_LIST: 0.9322690, DIVERSITY_HERFINDAHL: 0.9932256, COVERAGE_ITEM: 0.9995319, COVERAGE_ITEM_CORRECT: 0.0760228, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3179255, DIVERSITY_GINI: 0.2747563, SHANNON_ENTROPY: 9.9722220, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 195 of 1500. Elapsed time 41.56 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.45 seconds. BPR loss 7.19E-01. Sample per second: 15649\n",
      "MF_BPR: Epoch 196 of 1500. Elapsed time 41.63 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.56 seconds. BPR loss 7.31E-01. Sample per second: 15257\n",
      "MF_BPR: Epoch 197 of 1500. Elapsed time 41.70 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.80 seconds. BPR loss 7.49E-01. Sample per second: 14517\n",
      "MF_BPR: Epoch 198 of 1500. Elapsed time 41.77 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.95 seconds. BPR loss 7.66E-01. Sample per second: 14056\n",
      "MF_BPR: Epoch 199 of 1500. Elapsed time 41.84 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.03 seconds. BPR loss 7.92E-01. Sample per second: 13841\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.54 sec. Users per second: 1637\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.57 sec. Users per second: 1636\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1837552, PRECISION: 0.0603102, PRECISION_RECALL_MIN_DEN: 0.0787720, RECALL: 0.0547458, MAP: 0.0359902, MRR: 0.1484092, NDCG: 0.0496447, F1: 0.0573935, HIT_RATE: 0.6031019, ARHR: 0.2009320, NOVELTY: 0.0107929, AVERAGE_POPULARITY: 0.3968074, DIVERSITY_MEAN_INTER_LIST: 0.9285770, DIVERSITY_HERFINDAHL: 0.9928564, COVERAGE_ITEM: 0.9991574, COVERAGE_ITEM_CORRECT: 0.0755547, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3244655, DIVERSITY_GINI: 0.2643691, SHANNON_ENTROPY: 9.8594725, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 200 of 1500. Elapsed time 42.62 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.80 seconds. BPR loss 8.14E-01. Sample per second: 14520\n",
      "MF_BPR: Epoch 201 of 1500. Elapsed time 42.69 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.12 seconds. BPR loss 8.44E-01. Sample per second: 13608\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 202 of 1500. Elapsed time 42.76 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.32 seconds. BPR loss 8.45E-01. Sample per second: 16118\n",
      "MF_BPR: Epoch 203 of 1500. Elapsed time 42.83 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.41 seconds. BPR loss 8.74E-01. Sample per second: 15784\n",
      "MF_BPR: Epoch 204 of 1500. Elapsed time 42.90 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.46 seconds. BPR loss 8.82E-01. Sample per second: 15600\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.46 sec. Users per second: 1707\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.32 sec. Users per second: 1727\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1869120, PRECISION: 0.0616773, PRECISION_RECALL_MIN_DEN: 0.0803804, RECALL: 0.0556187, MAP: 0.0367979, MRR: 0.1513608, NDCG: 0.0506410, F1: 0.0584915, HIT_RATE: 0.6167732, ARHR: 0.2053235, NOVELTY: 0.0107047, AVERAGE_POPULARITY: 0.4061921, DIVERSITY_MEAN_INTER_LIST: 0.9250051, DIVERSITY_HERFINDAHL: 0.9924992, COVERAGE_ITEM: 0.9991574, COVERAGE_ITEM_CORRECT: 0.0749930, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3304044, DIVERSITY_GINI: 0.2548497, SHANNON_ENTROPY: 9.7539347, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 205 of 1500. Elapsed time 43.64 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.39 seconds. BPR loss 8.94E-01. Sample per second: 12916\n",
      "MF_BPR: Epoch 206 of 1500. Elapsed time 43.72 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.46 seconds. BPR loss 9.34E-01. Sample per second: 15610\n",
      "MF_BPR: Epoch 207 of 1500. Elapsed time 43.78 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.57 seconds. BPR loss 9.64E-01. Sample per second: 15241\n",
      "MF_BPR: Epoch 208 of 1500. Elapsed time 43.85 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.72 seconds. BPR loss 9.68E-01. Sample per second: 14753\n",
      "MF_BPR: Epoch 209 of 1500. Elapsed time 43.92 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.87 seconds. BPR loss 9.97E-01. Sample per second: 14297\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.20 sec. Users per second: 1755\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.60 sec. Users per second: 1759\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1896829, PRECISION: 0.0628879, PRECISION_RECALL_MIN_DEN: 0.0817907, RECALL: 0.0563663, MAP: 0.0374489, MRR: 0.1537563, NDCG: 0.0514608, F1: 0.0594488, HIT_RATE: 0.6288792, ARHR: 0.2090100, NOVELTY: 0.0106244, AVERAGE_POPULARITY: 0.4147253, DIVERSITY_MEAN_INTER_LIST: 0.9217523, DIVERSITY_HERFINDAHL: 0.9921739, COVERAGE_ITEM: 0.9990638, COVERAGE_ITEM_CORRECT: 0.0736822, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3358568, DIVERSITY_GINI: 0.2463029, SHANNON_ENTROPY: 9.6579763, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 210 of 1500. Elapsed time 44.65 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.08 seconds. BPR loss 1.03E+00. Sample per second: 13704\n",
      "MF_BPR: Epoch 211 of 1500. Elapsed time 44.73 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.26 seconds. BPR loss 1.04E+00. Sample per second: 16330\n",
      "MF_BPR: Epoch 212 of 1500. Elapsed time 44.80 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.32 seconds. BPR loss 1.06E+00. Sample per second: 16134\n",
      "MF_BPR: Epoch 213 of 1500. Elapsed time 44.86 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.32 seconds. BPR loss 1.09E+00. Sample per second: 16114\n",
      "MF_BPR: Epoch 214 of 1500. Elapsed time 44.93 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.36 seconds. BPR loss 1.09E+00. Sample per second: 15957\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.35 sec. Users per second: 1713\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.88 sec. Users per second: 1703\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1925324, PRECISION: 0.0640569, PRECISION_RECALL_MIN_DEN: 0.0831372, RECALL: 0.0570706, MAP: 0.0381129, MRR: 0.1562612, NDCG: 0.0522414, F1: 0.0603623, HIT_RATE: 0.6405687, ARHR: 0.2127125, NOVELTY: 0.0105502, AVERAGE_POPULARITY: 0.4229582, DIVERSITY_MEAN_INTER_LIST: 0.9185514, DIVERSITY_HERFINDAHL: 0.9918538, COVERAGE_ITEM: 0.9990638, COVERAGE_ITEM_CORRECT: 0.0719970, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3411947, DIVERSITY_GINI: 0.2385001, SHANNON_ENTROPY: 9.5685911, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 215 of 1500. Elapsed time 45.68 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.41 seconds. BPR loss 1.13E+00. Sample per second: 15801\n",
      "MF_BPR: Epoch 216 of 1500. Elapsed time 45.75 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.51 seconds. BPR loss 1.13E+00. Sample per second: 15452\n",
      "MF_BPR: Epoch 217 of 1500. Elapsed time 45.82 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.56 seconds. BPR loss 1.15E+00. Sample per second: 15265\n",
      "MF_BPR: Epoch 218 of 1500. Elapsed time 45.89 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.58 seconds. BPR loss 1.19E+00. Sample per second: 15206\n",
      "MF_BPR: Epoch 219 of 1500. Elapsed time 45.95 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.71 seconds. BPR loss 1.22E+00. Sample per second: 14788\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.22 sec. Users per second: 1754\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.41 sec. Users per second: 1767\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1950339, PRECISION: 0.0652115, PRECISION_RECALL_MIN_DEN: 0.0844881, RECALL: 0.0577848, MAP: 0.0386966, MRR: 0.1583124, NDCG: 0.0529909, F1: 0.0612739, HIT_RATE: 0.6521146, ARHR: 0.2159895, NOVELTY: 0.0104824, AVERAGE_POPULARITY: 0.4305427, DIVERSITY_MEAN_INTER_LIST: 0.9155460, DIVERSITY_HERFINDAHL: 0.9915533, COVERAGE_ITEM: 0.9989701, COVERAGE_ITEM_CORRECT: 0.0708735, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3463465, DIVERSITY_GINI: 0.2312030, SHANNON_ENTROPY: 9.4859039, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 220 of 1500. Elapsed time 46.68 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.54 seconds. BPR loss 1.26E+00. Sample per second: 15348\n",
      "MF_BPR: Epoch 221 of 1500. Elapsed time 46.75 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.69 seconds. BPR loss 1.24E+00. Sample per second: 14835\n",
      "MF_BPR: Epoch 222 of 1500. Elapsed time 46.82 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.93 seconds. BPR loss 1.30E+00. Sample per second: 14111\n",
      "MF_BPR: Epoch 223 of 1500. Elapsed time 46.89 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.93 seconds. BPR loss 1.31E+00. Sample per second: 14115\n",
      "MF_BPR: Epoch 224 of 1500. Elapsed time 46.96 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.05 seconds. BPR loss 1.33E+00. Sample per second: 13782\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.11 sec. Users per second: 1826\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.22 sec. Users per second: 1822\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1971982, PRECISION: 0.0662224, PRECISION_RECALL_MIN_DEN: 0.0857226, RECALL: 0.0584831, MAP: 0.0392722, MRR: 0.1604184, NDCG: 0.0536787, F1: 0.0621126, HIT_RATE: 0.6622245, ARHR: 0.2191861, NOVELTY: 0.0104205, AVERAGE_POPULARITY: 0.4375482, DIVERSITY_MEAN_INTER_LIST: 0.9126815, DIVERSITY_HERFINDAHL: 0.9912668, COVERAGE_ITEM: 0.9988765, COVERAGE_ITEM_CORRECT: 0.0693755, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3503392, DIVERSITY_GINI: 0.2245667, SHANNON_ENTROPY: 9.4089638, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 225 of 1500. Elapsed time 47.67 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.66 seconds. BPR loss 1.38E+00. Sample per second: 14943\n",
      "MF_BPR: Epoch 226 of 1500. Elapsed time 47.74 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.86 seconds. BPR loss 1.38E+00. Sample per second: 14334\n",
      "MF_BPR: Epoch 227 of 1500. Elapsed time 47.81 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.98 seconds. BPR loss 1.41E+00. Sample per second: 13979\n",
      "MF_BPR: Epoch 228 of 1500. Elapsed time 47.88 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.01 seconds. BPR loss 1.44E+00. Sample per second: 13887\n",
      "MF_BPR: Epoch 229 of 1500. Elapsed time 47.94 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.97 seconds. BPR loss 1.46E+00. Sample per second: 17539\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.15 sec. Users per second: 1758\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.22 sec. Users per second: 1776\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1997101, PRECISION: 0.0671674, PRECISION_RECALL_MIN_DEN: 0.0868275, RECALL: 0.0590412, MAP: 0.0398140, MRR: 0.1624546, NDCG: 0.0543860, F1: 0.0628427, HIT_RATE: 0.6716737, ARHR: 0.2222541, NOVELTY: 0.0103624, AVERAGE_POPULARITY: 0.4441724, DIVERSITY_MEAN_INTER_LIST: 0.9100746, DIVERSITY_HERFINDAHL: 0.9910062, COVERAGE_ITEM: 0.9987829, COVERAGE_ITEM_CORRECT: 0.0682520, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3544606, DIVERSITY_GINI: 0.2184470, SHANNON_ENTROPY: 9.3378873, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 230 of 1500. Elapsed time 48.66 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.25 seconds. BPR loss 1.50E+00. Sample per second: 16376\n",
      "MF_BPR: Epoch 231 of 1500. Elapsed time 48.73 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.22 seconds. BPR loss 1.48E+00. Sample per second: 16516\n",
      "MF_BPR: Epoch 232 of 1500. Elapsed time 48.80 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.18 seconds. BPR loss 1.53E+00. Sample per second: 16677\n",
      "MF_BPR: Epoch 233 of 1500. Elapsed time 48.86 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.15 seconds. BPR loss 1.54E+00. Sample per second: 16774\n",
      "MF_BPR: Epoch 234 of 1500. Elapsed time 48.93 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.13 seconds. BPR loss 1.60E+00. Sample per second: 16856\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.32 sec. Users per second: 1781\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.50 sec. Users per second: 1763\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2020307, PRECISION: 0.0680520, PRECISION_RECALL_MIN_DEN: 0.0878281, RECALL: 0.0595702, MAP: 0.0402875, MRR: 0.1643686, NDCG: 0.0550706, F1: 0.0635292, HIT_RATE: 0.6805199, ARHR: 0.2251171, NOVELTY: 0.0103103, AVERAGE_POPULARITY: 0.4503607, DIVERSITY_MEAN_INTER_LIST: 0.9076545, DIVERSITY_HERFINDAHL: 0.9907641, COVERAGE_ITEM: 0.9988765, COVERAGE_ITEM_CORRECT: 0.0673158, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3582673, DIVERSITY_GINI: 0.2129098, SHANNON_ENTROPY: 9.2739702, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 235 of 1500. Elapsed time 49.65 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.82 seconds. BPR loss 1.61E+00. Sample per second: 14455\n",
      "MF_BPR: Epoch 236 of 1500. Elapsed time 49.72 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.88 seconds. BPR loss 1.65E+00. Sample per second: 14281\n",
      "MF_BPR: Epoch 237 of 1500. Elapsed time 49.79 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.89 seconds. BPR loss 1.68E+00. Sample per second: 14247\n",
      "MF_BPR: Epoch 238 of 1500. Elapsed time 49.86 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.86 seconds. BPR loss 1.71E+00. Sample per second: 14325\n",
      "MF_BPR: Epoch 239 of 1500. Elapsed time 49.92 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.89 seconds. BPR loss 1.73E+00. Sample per second: 14227\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.51 sec. Users per second: 1770\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.49 sec. Users per second: 1763\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2040629, PRECISION: 0.0690170, PRECISION_RECALL_MIN_DEN: 0.0889306, RECALL: 0.0601055, MAP: 0.0408172, MRR: 0.1666053, NDCG: 0.0558106, F1: 0.0642538, HIT_RATE: 0.6901702, ARHR: 0.2283677, NOVELTY: 0.0102631, AVERAGE_POPULARITY: 0.4562890, DIVERSITY_MEAN_INTER_LIST: 0.9054128, DIVERSITY_HERFINDAHL: 0.9905400, COVERAGE_ITEM: 0.9985956, COVERAGE_ITEM_CORRECT: 0.0665668, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3624603, DIVERSITY_GINI: 0.2080008, SHANNON_ENTROPY: 9.2170283, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 240 of 1500. Elapsed time 50.65 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.53 seconds. BPR loss 1.76E+00. Sample per second: 15380\n",
      "MF_BPR: Epoch 241 of 1500. Elapsed time 50.72 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.28 seconds. BPR loss 1.79E+00. Sample per second: 13192\n",
      "MF_BPR: Epoch 242 of 1500. Elapsed time 50.80 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.33 seconds. BPR loss 1.82E+00. Sample per second: 16097\n",
      "MF_BPR: Epoch 243 of 1500. Elapsed time 50.86 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.38 seconds. BPR loss 1.85E+00. Sample per second: 15887\n",
      "MF_BPR: Epoch 244 of 1500. Elapsed time 50.93 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 1.90E+00. Sample per second: 15917\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.03 sec. Users per second: 1765\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.24 sec. Users per second: 1775\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2060586, PRECISION: 0.0698327, PRECISION_RECALL_MIN_DEN: 0.0898677, RECALL: 0.0605600, MAP: 0.0412972, MRR: 0.1682410, NDCG: 0.0564792, F1: 0.0648666, HIT_RATE: 0.6983270, ARHR: 0.2309696, NOVELTY: 0.0102200, AVERAGE_POPULARITY: 0.4619237, DIVERSITY_MEAN_INTER_LIST: 0.9030735, DIVERSITY_HERFINDAHL: 0.9903061, COVERAGE_ITEM: 0.9984084, COVERAGE_ITEM_CORRECT: 0.0657242, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3657804, DIVERSITY_GINI: 0.2035992, SHANNON_ENTROPY: 9.1631045, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 245 of 1500. Elapsed time 51.65 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.71 seconds. BPR loss 1.91E+00. Sample per second: 14780\n",
      "MF_BPR: Epoch 246 of 1500. Elapsed time 51.72 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.48 seconds. BPR loss 1.94E+00. Sample per second: 12696\n",
      "MF_BPR: Epoch 247 of 1500. Elapsed time 51.80 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.49 seconds. BPR loss 1.94E+00. Sample per second: 15513\n",
      "MF_BPR: Epoch 248 of 1500. Elapsed time 51.87 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.50 seconds. BPR loss 1.98E+00. Sample per second: 15481\n",
      "MF_BPR: Epoch 249 of 1500. Elapsed time 51.93 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.61 seconds. BPR loss 2.01E+00. Sample per second: 15117\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.34 sec. Users per second: 1714\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.85 sec. Users per second: 1748\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2076846, PRECISION: 0.0706039, PRECISION_RECALL_MIN_DEN: 0.0907376, RECALL: 0.0609726, MAP: 0.0417375, MRR: 0.1699510, NDCG: 0.0570641, F1: 0.0654357, HIT_RATE: 0.7060386, ARHR: 0.2336348, NOVELTY: 0.0101805, AVERAGE_POPULARITY: 0.4670546, DIVERSITY_MEAN_INTER_LIST: 0.9009124, DIVERSITY_HERFINDAHL: 0.9900900, COVERAGE_ITEM: 0.9983148, COVERAGE_ITEM_CORRECT: 0.0652561, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3684565, DIVERSITY_GINI: 0.1995496, SHANNON_ENTROPY: 9.1137777, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 250 of 1500. Elapsed time 52.67 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.61 seconds. BPR loss 2.05E+00. Sample per second: 15107\n",
      "MF_BPR: Epoch 251 of 1500. Elapsed time 52.74 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.81 seconds. BPR loss 2.10E+00. Sample per second: 14485\n",
      "MF_BPR: Epoch 252 of 1500. Elapsed time 52.81 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.55 seconds. BPR loss 2.09E+00. Sample per second: 15319\n",
      "MF_BPR: Epoch 253 of 1500. Elapsed time 52.87 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.25 seconds. BPR loss 2.13E+00. Sample per second: 16392\n",
      "MF_BPR: Epoch 254 of 1500. Elapsed time 52.93 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.09 seconds. BPR loss 2.15E+00. Sample per second: 17018\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 61000 ( 87.60% ) in 30.36 sec. Users per second: 2009\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 34.59 sec. Users per second: 2013\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2093468, PRECISION: 0.0713492, PRECISION_RECALL_MIN_DEN: 0.0915228, RECALL: 0.0613260, MAP: 0.0421098, MRR: 0.1715540, NDCG: 0.0576229, F1: 0.0659590, HIT_RATE: 0.7134918, ARHR: 0.2360313, NOVELTY: 0.0101440, AVERAGE_POPULARITY: 0.4719722, DIVERSITY_MEAN_INTER_LIST: 0.8988042, DIVERSITY_HERFINDAHL: 0.9898791, COVERAGE_ITEM: 0.9982211, COVERAGE_ITEM_CORRECT: 0.0640389, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3715905, DIVERSITY_GINI: 0.1957356, SHANNON_ENTROPY: 9.0675453, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 255 of 1500. Elapsed time 53.57 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.50 seconds. BPR loss 2.19E+00. Sample per second: 15460\n",
      "MF_BPR: Epoch 256 of 1500. Elapsed time 53.63 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.22 seconds. BPR loss 2.25E+00. Sample per second: 16512\n",
      "MF_BPR: Epoch 257 of 1500. Elapsed time 53.70 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.95 seconds. BPR loss 2.23E+00. Sample per second: 17647\n",
      "MF_BPR: Epoch 258 of 1500. Elapsed time 53.76 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.68 seconds. BPR loss 2.28E+00. Sample per second: 14882\n",
      "MF_BPR: Epoch 259 of 1500. Elapsed time 53.82 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 2.31E+00. Sample per second: 15923\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 61000 ( 87.60% ) in 30.41 sec. Users per second: 2006\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 34.66 sec. Users per second: 2009\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2111346, PRECISION: 0.0719408, PRECISION_RECALL_MIN_DEN: 0.0921732, RECALL: 0.0616386, MAP: 0.0424645, MRR: 0.1730183, NDCG: 0.0581800, F1: 0.0663924, HIT_RATE: 0.7194083, ARHR: 0.2382140, NOVELTY: 0.0101111, AVERAGE_POPULARITY: 0.4764516, DIVERSITY_MEAN_INTER_LIST: 0.8969153, DIVERSITY_HERFINDAHL: 0.9896902, COVERAGE_ITEM: 0.9982211, COVERAGE_ITEM_CORRECT: 0.0635708, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3742380, DIVERSITY_GINI: 0.1923931, SHANNON_ENTROPY: 9.0270133, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 260 of 1500. Elapsed time 54.46 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.92 seconds. BPR loss 2.34E+00. Sample per second: 17775\n",
      "MF_BPR: Epoch 261 of 1500. Elapsed time 54.52 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.57 seconds. BPR loss 2.38E+00. Sample per second: 15253\n",
      "MF_BPR: Epoch 262 of 1500. Elapsed time 54.59 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.25 seconds. BPR loss 2.41E+00. Sample per second: 16387\n",
      "MF_BPR: Epoch 263 of 1500. Elapsed time 54.65 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.91 seconds. BPR loss 2.43E+00. Sample per second: 17829\n",
      "MF_BPR: Epoch 264 of 1500. Elapsed time 54.71 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.57 seconds. BPR loss 2.45E+00. Sample per second: 15229\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 60000 ( 86.16% ) in 30.01 sec. Users per second: 1999\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 34.74 sec. Users per second: 2004\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2128737, PRECISION: 0.0725641, PRECISION_RECALL_MIN_DEN: 0.0929069, RECALL: 0.0620084, MAP: 0.0427886, MRR: 0.1743387, NDCG: 0.0586945, F1: 0.0668722, HIT_RATE: 0.7256408, ARHR: 0.2401698, NOVELTY: 0.0100804, AVERAGE_POPULARITY: 0.4807869, DIVERSITY_MEAN_INTER_LIST: 0.8949798, DIVERSITY_HERFINDAHL: 0.9894967, COVERAGE_ITEM: 0.9980339, COVERAGE_ITEM_CORRECT: 0.0622601, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3771573, DIVERSITY_GINI: 0.1891581, SHANNON_ENTROPY: 8.9880247, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 265 of 1500. Elapsed time 55.35 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.10 seconds. BPR loss 2.52E+00. Sample per second: 16977\n",
      "MF_BPR: Epoch 266 of 1500. Elapsed time 55.41 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.79 seconds. BPR loss 2.51E+00. Sample per second: 18391\n",
      "MF_BPR: Epoch 267 of 1500. Elapsed time 55.47 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.45 seconds. BPR loss 2.59E+00. Sample per second: 15638\n",
      "MF_BPR: Epoch 268 of 1500. Elapsed time 55.53 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.16 seconds. BPR loss 2.58E+00. Sample per second: 16722\n",
      "MF_BPR: Epoch 269 of 1500. Elapsed time 55.60 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.94 seconds. BPR loss 2.60E+00. Sample per second: 17676\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.14 sec. Users per second: 1957\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.55 sec. Users per second: 1959\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2143049, PRECISION: 0.0731974, PRECISION_RECALL_MIN_DEN: 0.0936371, RECALL: 0.0623926, MAP: 0.0431015, MRR: 0.1757793, NDCG: 0.0592128, F1: 0.0673645, HIT_RATE: 0.7319739, ARHR: 0.2422455, NOVELTY: 0.0100527, AVERAGE_POPULARITY: 0.4849687, DIVERSITY_MEAN_INTER_LIST: 0.8931200, DIVERSITY_HERFINDAHL: 0.9893107, COVERAGE_ITEM: 0.9980339, COVERAGE_ITEM_CORRECT: 0.0617920, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3800767, DIVERSITY_GINI: 0.1862967, SHANNON_ENTROPY: 8.9527400, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 270 of 1500. Elapsed time 56.25 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.36 seconds. BPR loss 2.68E+00. Sample per second: 15965\n",
      "MF_BPR: Epoch 271 of 1500. Elapsed time 56.32 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.07 seconds. BPR loss 2.73E+00. Sample per second: 17090\n",
      "MF_BPR: Epoch 272 of 1500. Elapsed time 56.38 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.81 seconds. BPR loss 2.72E+00. Sample per second: 18256\n",
      "MF_BPR: Epoch 273 of 1500. Elapsed time 56.44 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.46 seconds. BPR loss 2.78E+00. Sample per second: 15611\n",
      "MF_BPR: Epoch 274 of 1500. Elapsed time 56.50 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.17 seconds. BPR loss 2.79E+00. Sample per second: 16715\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 61000 ( 87.60% ) in 30.04 sec. Users per second: 2031\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 34.22 sec. Users per second: 2035\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2155251, PRECISION: 0.0738149, PRECISION_RECALL_MIN_DEN: 0.0943690, RECALL: 0.0627403, MAP: 0.0434118, MRR: 0.1769919, NDCG: 0.0596652, F1: 0.0678285, HIT_RATE: 0.7381489, ARHR: 0.2440931, NOVELTY: 0.0100277, AVERAGE_POPULARITY: 0.4886680, DIVERSITY_MEAN_INTER_LIST: 0.8914276, DIVERSITY_HERFINDAHL: 0.9891415, COVERAGE_ITEM: 0.9976594, COVERAGE_ITEM_CORRECT: 0.0616983, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3829818, DIVERSITY_GINI: 0.1837949, SHANNON_ENTROPY: 8.9213175, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 275 of 1500. Elapsed time 57.13 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.13 seconds. BPR loss 2.79E+00. Sample per second: 16856\n",
      "MF_BPR: Epoch 276 of 1500. Elapsed time 57.19 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.80 seconds. BPR loss 2.85E+00. Sample per second: 18324\n",
      "MF_BPR: Epoch 277 of 1500. Elapsed time 57.26 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.51 seconds. BPR loss 2.89E+00. Sample per second: 15447\n",
      "MF_BPR: Epoch 278 of 1500. Elapsed time 57.32 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.39 seconds. BPR loss 2.93E+00. Sample per second: 15865\n",
      "MF_BPR: Epoch 279 of 1500. Elapsed time 57.38 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.04 seconds. BPR loss 2.94E+00. Sample per second: 17250\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 29000 ( 41.65% ) in 34.04 sec. Users per second: 852\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 55.14 sec. Users per second: 1263\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2169178, PRECISION: 0.0742888, PRECISION_RECALL_MIN_DEN: 0.0949346, RECALL: 0.0630078, MAP: 0.0436868, MRR: 0.1780918, NDCG: 0.0600892, F1: 0.0681849, HIT_RATE: 0.7428879, ARHR: 0.2458081, NOVELTY: 0.0100049, AVERAGE_POPULARITY: 0.4919427, DIVERSITY_MEAN_INTER_LIST: 0.8900438, DIVERSITY_HERFINDAHL: 0.9890031, COVERAGE_ITEM: 0.9974721, COVERAGE_ITEM_CORRECT: 0.0609493, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3848851, DIVERSITY_GINI: 0.1814387, SHANNON_ENTROPY: 8.8939806, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 280 of 1500. Elapsed time 58.36 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.19 seconds. BPR loss 2.99E+00. Sample per second: 16633\n",
      "MF_BPR: Epoch 281 of 1500. Elapsed time 58.43 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.10 seconds. BPR loss 3.01E+00. Sample per second: 17004\n",
      "MF_BPR: Epoch 282 of 1500. Elapsed time 58.49 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.89 seconds. BPR loss 3.04E+00. Sample per second: 17900\n",
      "MF_BPR: Epoch 283 of 1500. Elapsed time 58.56 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.74 seconds. BPR loss 3.07E+00. Sample per second: 14703\n",
      "MF_BPR: Epoch 284 of 1500. Elapsed time 58.62 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.47 seconds. BPR loss 3.15E+00. Sample per second: 15591\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 58000 ( 83.29% ) in 30.26 sec. Users per second: 1917\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 36.17 sec. Users per second: 1925\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2181187, PRECISION: 0.0748991, PRECISION_RECALL_MIN_DEN: 0.0956401, RECALL: 0.0633836, MAP: 0.0439975, MRR: 0.1793824, NDCG: 0.0606133, F1: 0.0686619, HIT_RATE: 0.7489912, ARHR: 0.2477259, NOVELTY: 0.0099828, AVERAGE_POPULARITY: 0.4952375, DIVERSITY_MEAN_INTER_LIST: 0.8886014, DIVERSITY_HERFINDAHL: 0.9888589, COVERAGE_ITEM: 0.9973785, COVERAGE_ITEM_CORRECT: 0.0607621, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3874753, DIVERSITY_GINI: 0.1792577, SHANNON_ENTROPY: 8.8681245, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 285 of 1500. Elapsed time 59.29 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.52 seconds. BPR loss 3.17E+00. Sample per second: 15394\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 286 of 1500. Elapsed time 59.35 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.24 seconds. BPR loss 3.21E+00. Sample per second: 16413\n",
      "MF_BPR: Epoch 287 of 1500. Elapsed time 59.41 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.07 seconds. BPR loss 3.26E+00. Sample per second: 17095\n",
      "MF_BPR: Epoch 288 of 1500. Elapsed time 59.48 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.87 seconds. BPR loss 3.23E+00. Sample per second: 18001\n",
      "MF_BPR: Epoch 289 of 1500. Elapsed time 59.54 min\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.69 seconds. BPR loss 3.30E+00. Sample per second: 14834\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.18 sec. Users per second: 1955\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.51 sec. Users per second: 1961\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2192077, PRECISION: 0.0753141, PRECISION_RECALL_MIN_DEN: 0.0960584, RECALL: 0.0635302, MAP: 0.0442067, MRR: 0.1802587, NDCG: 0.0609338, F1: 0.0689221, HIT_RATE: 0.7531414, ARHR: 0.2490990, NOVELTY: 0.0099618, AVERAGE_POPULARITY: 0.4983451, DIVERSITY_MEAN_INTER_LIST: 0.8871211, DIVERSITY_HERFINDAHL: 0.9887108, COVERAGE_ITEM: 0.9970977, COVERAGE_ITEM_CORRECT: 0.0596386, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3890066, DIVERSITY_GINI: 0.1770610, SHANNON_ENTROPY: 8.8421677, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 290 of 1500. Elapsed time 1.00 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.08 seconds. BPR loss 3.34E+00. Sample per second: 17070\n",
      "MF_BPR: Epoch 291 of 1500. Elapsed time 1.00 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.90 seconds. BPR loss 3.38E+00. Sample per second: 17835\n",
      "MF_BPR: Epoch 292 of 1500. Elapsed time 1.01 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.72 seconds. BPR loss 3.36E+00. Sample per second: 14760\n",
      "MF_BPR: Epoch 293 of 1500. Elapsed time 1.01 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.56 seconds. BPR loss 3.41E+00. Sample per second: 15276\n",
      "MF_BPR: Epoch 294 of 1500. Elapsed time 1.01 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.43 seconds. BPR loss 3.45E+00. Sample per second: 15734\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.39 sec. Users per second: 1941\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.90 sec. Users per second: 1940\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2205122, PRECISION: 0.0757004, PRECISION_RECALL_MIN_DEN: 0.0964505, RECALL: 0.0636675, MAP: 0.0444014, MRR: 0.1812896, NDCG: 0.0612713, F1: 0.0691645, HIT_RATE: 0.7570044, ARHR: 0.2505601, NOVELTY: 0.0099431, AVERAGE_POPULARITY: 0.5013044, DIVERSITY_MEAN_INTER_LIST: 0.8857545, DIVERSITY_HERFINDAHL: 0.9885742, COVERAGE_ITEM: 0.9968168, COVERAGE_ITEM_CORRECT: 0.0593577, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3909385, DIVERSITY_GINI: 0.1751727, SHANNON_ENTROPY: 8.8193985, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 295 of 1500. Elapsed time 1.02 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.14 seconds. BPR loss 3.49E+00. Sample per second: 16820\n",
      "MF_BPR: Epoch 296 of 1500. Elapsed time 1.02 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.88 seconds. BPR loss 3.52E+00. Sample per second: 17934\n",
      "MF_BPR: Epoch 297 of 1500. Elapsed time 1.02 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.67 seconds. BPR loss 3.52E+00. Sample per second: 14911\n",
      "MF_BPR: Epoch 298 of 1500. Elapsed time 1.02 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.39 seconds. BPR loss 3.59E+00. Sample per second: 15877\n",
      "MF_BPR: Epoch 299 of 1500. Elapsed time 1.02 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.19 seconds. BPR loss 3.66E+00. Sample per second: 16612\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 58000 ( 83.29% ) in 30.08 sec. Users per second: 1928\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 36.14 sec. Users per second: 1927\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2211847, PRECISION: 0.0762232, PRECISION_RECALL_MIN_DEN: 0.0969728, RECALL: 0.0638996, MAP: 0.0446261, MRR: 0.1822463, NDCG: 0.0616602, F1: 0.0695195, HIT_RATE: 0.7622316, ARHR: 0.2521245, NOVELTY: 0.0099265, AVERAGE_POPULARITY: 0.5039674, DIVERSITY_MEAN_INTER_LIST: 0.8845336, DIVERSITY_HERFINDAHL: 0.9884521, COVERAGE_ITEM: 0.9967232, COVERAGE_ITEM_CORRECT: 0.0584215, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3927989, DIVERSITY_GINI: 0.1735181, SHANNON_ENTROPY: 8.7998861, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 300 of 1500. Elapsed time 1.03 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.28 seconds. BPR loss 3.67E+00. Sample per second: 16252\n",
      "MF_BPR: Epoch 301 of 1500. Elapsed time 1.03 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.09 seconds. BPR loss 3.75E+00. Sample per second: 17034\n",
      "MF_BPR: Epoch 302 of 1500. Elapsed time 1.04 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.84 seconds. BPR loss 3.72E+00. Sample per second: 18131\n",
      "MF_BPR: Epoch 303 of 1500. Elapsed time 1.04 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.62 seconds. BPR loss 3.75E+00. Sample per second: 15078\n",
      "MF_BPR: Epoch 304 of 1500. Elapsed time 1.04 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.36 seconds. BPR loss 3.75E+00. Sample per second: 15959\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.43 sec. Users per second: 1939\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.69 sec. Users per second: 1951\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2217716, PRECISION: 0.0766166, PRECISION_RECALL_MIN_DEN: 0.0974289, RECALL: 0.0640895, MAP: 0.0447898, MRR: 0.1828532, NDCG: 0.0620149, F1: 0.0697954, HIT_RATE: 0.7661664, ARHR: 0.2531258, NOVELTY: 0.0099108, AVERAGE_POPULARITY: 0.5062398, DIVERSITY_MEAN_INTER_LIST: 0.8835917, DIVERSITY_HERFINDAHL: 0.9883579, COVERAGE_ITEM: 0.9966295, COVERAGE_ITEM_CORRECT: 0.0586087, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3942156, DIVERSITY_GINI: 0.1718782, SHANNON_ENTROPY: 8.7826196, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 305 of 1500. Elapsed time 1.05 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.90 seconds. BPR loss 3.90E+00. Sample per second: 17838\n",
      "MF_BPR: Epoch 306 of 1500. Elapsed time 1.05 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.61 seconds. BPR loss 3.86E+00. Sample per second: 15112\n",
      "MF_BPR: Epoch 307 of 1500. Elapsed time 1.05 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.34 seconds. BPR loss 3.88E+00. Sample per second: 16057\n",
      "MF_BPR: Epoch 308 of 1500. Elapsed time 1.05 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.01 seconds. BPR loss 3.98E+00. Sample per second: 17367\n",
      "MF_BPR: Epoch 309 of 1500. Elapsed time 1.05 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.72 seconds. BPR loss 3.94E+00. Sample per second: 18697\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 58000 ( 83.29% ) in 30.28 sec. Users per second: 1915\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 36.11 sec. Users per second: 1929\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2227847, PRECISION: 0.0769096, PRECISION_RECALL_MIN_DEN: 0.0977173, RECALL: 0.0641509, MAP: 0.0449474, MRR: 0.1836652, NDCG: 0.0622941, F1: 0.0699533, HIT_RATE: 0.7690960, ARHR: 0.2542892, NOVELTY: 0.0098966, AVERAGE_POPULARITY: 0.5085039, DIVERSITY_MEAN_INTER_LIST: 0.8825412, DIVERSITY_HERFINDAHL: 0.9882529, COVERAGE_ITEM: 0.9967232, COVERAGE_ITEM_CORRECT: 0.0585151, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3954607, DIVERSITY_GINI: 0.1704320, SHANNON_ENTROPY: 8.7659629, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 310 of 1500. Elapsed time 1.06 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.71 seconds. BPR loss 4.04E+00. Sample per second: 14770\n",
      "MF_BPR: Epoch 311 of 1500. Elapsed time 1.07 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.48 seconds. BPR loss 4.07E+00. Sample per second: 15537\n",
      "MF_BPR: Epoch 312 of 1500. Elapsed time 1.07 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.27 seconds. BPR loss 4.09E+00. Sample per second: 16293\n",
      "MF_BPR: Epoch 313 of 1500. Elapsed time 1.07 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.98 seconds. BPR loss 4.16E+00. Sample per second: 17498\n",
      "MF_BPR: Epoch 314 of 1500. Elapsed time 1.07 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.69 seconds. BPR loss 4.20E+00. Sample per second: 14856\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.52 sec. Users per second: 1933\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 36.00 sec. Users per second: 1934\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2235847, PRECISION: 0.0772586, PRECISION_RECALL_MIN_DEN: 0.0981170, RECALL: 0.0643168, MAP: 0.0451547, MRR: 0.1844031, NDCG: 0.0626750, F1: 0.0701962, HIT_RATE: 0.7725856, ARHR: 0.2554723, NOVELTY: 0.0098844, AVERAGE_POPULARITY: 0.5105496, DIVERSITY_MEAN_INTER_LIST: 0.8815702, DIVERSITY_HERFINDAHL: 0.9881558, COVERAGE_ITEM: 0.9963487, COVERAGE_ITEM_CORRECT: 0.0583279, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3970205, DIVERSITY_GINI: 0.1691112, SHANNON_ENTROPY: 8.7511115, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 315 of 1500. Elapsed time 1.08 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.53 seconds. BPR loss 4.17E+00. Sample per second: 15386\n",
      "MF_BPR: Epoch 316 of 1500. Elapsed time 1.08 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 4.20E+00. Sample per second: 15940\n",
      "MF_BPR: Epoch 317 of 1500. Elapsed time 1.08 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.15 seconds. BPR loss 4.25E+00. Sample per second: 16763\n",
      "MF_BPR: Epoch 318 of 1500. Elapsed time 1.08 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.92 seconds. BPR loss 4.31E+00. Sample per second: 17767\n",
      "MF_BPR: Epoch 319 of 1500. Elapsed time 1.08 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.61 seconds. BPR loss 4.33E+00. Sample per second: 15091\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 58000 ( 83.29% ) in 30.02 sec. Users per second: 1932\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.96 sec. Users per second: 1936\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2241630, PRECISION: 0.0776233, PRECISION_RECALL_MIN_DEN: 0.0985248, RECALL: 0.0644861, MAP: 0.0453251, MRR: 0.1850637, NDCG: 0.0629477, F1: 0.0704475, HIT_RATE: 0.7762332, ARHR: 0.2565689, NOVELTY: 0.0098733, AVERAGE_POPULARITY: 0.5123534, DIVERSITY_MEAN_INTER_LIST: 0.8807695, DIVERSITY_HERFINDAHL: 0.9880757, COVERAGE_ITEM: 0.9962550, COVERAGE_ITEM_CORRECT: 0.0581406, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3982369, DIVERSITY_GINI: 0.1678980, SHANNON_ENTROPY: 8.7384391, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 320 of 1500. Elapsed time 1.09 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.49 seconds. BPR loss 4.36E+00. Sample per second: 15509\n",
      "MF_BPR: Epoch 321 of 1500. Elapsed time 1.10 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.29 seconds. BPR loss 4.42E+00. Sample per second: 16215\n",
      "MF_BPR: Epoch 322 of 1500. Elapsed time 1.10 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.13 seconds. BPR loss 4.39E+00. Sample per second: 16878\n",
      "MF_BPR: Epoch 323 of 1500. Elapsed time 1.10 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.93 seconds. BPR loss 4.48E+00. Sample per second: 17715\n",
      "MF_BPR: Epoch 324 of 1500. Elapsed time 1.10 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.72 seconds. BPR loss 4.52E+00. Sample per second: 14760\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.46 sec. Users per second: 1937\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.80 sec. Users per second: 1945\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2248034, PRECISION: 0.0778488, PRECISION_RECALL_MIN_DEN: 0.0987748, RECALL: 0.0645991, MAP: 0.0454193, MRR: 0.1855080, NDCG: 0.0631444, F1: 0.0706077, HIT_RATE: 0.7784878, ARHR: 0.2572285, NOVELTY: 0.0098630, AVERAGE_POPULARITY: 0.5138670, DIVERSITY_MEAN_INTER_LIST: 0.8801787, DIVERSITY_HERFINDAHL: 0.9880166, COVERAGE_ITEM: 0.9961614, COVERAGE_ITEM_CORRECT: 0.0579534, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3994820, DIVERSITY_GINI: 0.1667218, SHANNON_ENTROPY: 8.7275368, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 325 of 1500. Elapsed time 1.11 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.27 seconds. BPR loss 4.51E+00. Sample per second: 16296\n",
      "MF_BPR: Epoch 326 of 1500. Elapsed time 1.11 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.02 seconds. BPR loss 4.58E+00. Sample per second: 17325\n",
      "MF_BPR: Epoch 327 of 1500. Elapsed time 1.11 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.79 seconds. BPR loss 4.60E+00. Sample per second: 18353\n",
      "MF_BPR: Epoch 328 of 1500. Elapsed time 1.11 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.58 seconds. BPR loss 4.69E+00. Sample per second: 15207\n",
      "MF_BPR: Epoch 329 of 1500. Elapsed time 1.11 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.38 seconds. BPR loss 4.69E+00. Sample per second: 15916\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 58000 ( 83.29% ) in 30.04 sec. Users per second: 1931\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.92 sec. Users per second: 1938\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2254157, PRECISION: 0.0780958, PRECISION_RECALL_MIN_DEN: 0.0990211, RECALL: 0.0646703, MAP: 0.0455057, MRR: 0.1858677, NDCG: 0.0633610, F1: 0.0707518, HIT_RATE: 0.7809579, ARHR: 0.2579102, NOVELTY: 0.0098540, AVERAGE_POPULARITY: 0.5152907, DIVERSITY_MEAN_INTER_LIST: 0.8797443, DIVERSITY_HERFINDAHL: 0.9879732, COVERAGE_ITEM: 0.9962550, COVERAGE_ITEM_CORRECT: 0.0576725, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4005266, DIVERSITY_GINI: 0.1655882, SHANNON_ENTROPY: 8.7186001, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 330 of 1500. Elapsed time 1.13 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.10 seconds. BPR loss 4.74E+00. Sample per second: 17003\n",
      "MF_BPR: Epoch 331 of 1500. Elapsed time 1.13 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.79 seconds. BPR loss 4.69E+00. Sample per second: 18375\n",
      "MF_BPR: Epoch 332 of 1500. Elapsed time 1.13 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.54 seconds. BPR loss 4.78E+00. Sample per second: 15349\n",
      "MF_BPR: Epoch 333 of 1500. Elapsed time 1.13 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.26 seconds. BPR loss 4.79E+00. Sample per second: 16344\n",
      "MF_BPR: Epoch 334 of 1500. Elapsed time 1.13 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.94 seconds. BPR loss 4.86E+00. Sample per second: 17683\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.50 sec. Users per second: 1934\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.89 sec. Users per second: 1940\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2258417, PRECISION: 0.0783988, PRECISION_RECALL_MIN_DEN: 0.0993320, RECALL: 0.0647630, MAP: 0.0456173, MRR: 0.1862555, NDCG: 0.0635202, F1: 0.0709315, HIT_RATE: 0.7839879, ARHR: 0.2586628, NOVELTY: 0.0098449, AVERAGE_POPULARITY: 0.5166447, DIVERSITY_MEAN_INTER_LIST: 0.8792045, DIVERSITY_HERFINDAHL: 0.9879192, COVERAGE_ITEM: 0.9962550, COVERAGE_ITEM_CORRECT: 0.0577661, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4015856, DIVERSITY_GINI: 0.1646457, SHANNON_ENTROPY: 8.7098130, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 335 of 1500. Elapsed time 1.14 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.69 seconds. BPR loss 4.84E+00. Sample per second: 14841\n",
      "MF_BPR: Epoch 336 of 1500. Elapsed time 1.14 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.49 seconds. BPR loss 4.95E+00. Sample per second: 15494\n",
      "MF_BPR: Epoch 337 of 1500. Elapsed time 1.14 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.19 seconds. BPR loss 4.99E+00. Sample per second: 16632\n",
      "MF_BPR: Epoch 338 of 1500. Elapsed time 1.14 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.81 seconds. BPR loss 4.98E+00. Sample per second: 18261\n",
      "MF_BPR: Epoch 339 of 1500. Elapsed time 1.14 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.44 seconds. BPR loss 5.03E+00. Sample per second: 15679\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 60000 ( 86.16% ) in 30.41 sec. Users per second: 1973\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.17 sec. Users per second: 1980\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2262256, PRECISION: 0.0786501, PRECISION_RECALL_MIN_DEN: 0.0995693, RECALL: 0.0648473, MAP: 0.0457380, MRR: 0.1867174, NDCG: 0.0637079, F1: 0.0710849, HIT_RATE: 0.7865010, ARHR: 0.2594108, NOVELTY: 0.0098376, AVERAGE_POPULARITY: 0.5175945, DIVERSITY_MEAN_INTER_LIST: 0.8790161, DIVERSITY_HERFINDAHL: 0.9879004, COVERAGE_ITEM: 0.9959742, COVERAGE_ITEM_CORRECT: 0.0577661, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4024013, DIVERSITY_GINI: 0.1637798, SHANNON_ENTROPY: 8.7033964, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 340 of 1500. Elapsed time 1.16 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.38 seconds. BPR loss 5.08E+00. Sample per second: 15890\n",
      "MF_BPR: Epoch 341 of 1500. Elapsed time 1.16 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.03 seconds. BPR loss 5.09E+00. Sample per second: 17269\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 342 of 1500. Elapsed time 1.16 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.80 seconds. BPR loss 5.11E+00. Sample per second: 18314\n",
      "MF_BPR: Epoch 343 of 1500. Elapsed time 1.16 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.59 seconds. BPR loss 5.17E+00. Sample per second: 15179\n",
      "MF_BPR: Epoch 344 of 1500. Elapsed time 1.16 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.25 seconds. BPR loss 5.16E+00. Sample per second: 16386\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.00 sec. Users per second: 1966\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.34 sec. Users per second: 1971\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2269207, PRECISION: 0.0787722, PRECISION_RECALL_MIN_DEN: 0.0997438, RECALL: 0.0649516, MAP: 0.0458097, MRR: 0.1871695, NDCG: 0.0638819, F1: 0.0711974, HIT_RATE: 0.7877217, ARHR: 0.2599994, NOVELTY: 0.0098300, AVERAGE_POPULARITY: 0.5186925, DIVERSITY_MEAN_INTER_LIST: 0.8786763, DIVERSITY_HERFINDAHL: 0.9878664, COVERAGE_ITEM: 0.9955060, COVERAGE_ITEM_CORRECT: 0.0577661, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4033172, DIVERSITY_GINI: 0.1628194, SHANNON_ENTROPY: 8.6967285, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 345 of 1500. Elapsed time 1.17 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.33 seconds. BPR loss 5.18E+00. Sample per second: 16096\n",
      "MF_BPR: Epoch 346 of 1500. Elapsed time 1.17 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.05 seconds. BPR loss 5.25E+00. Sample per second: 17195\n",
      "MF_BPR: Epoch 347 of 1500. Elapsed time 1.17 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.79 seconds. BPR loss 5.34E+00. Sample per second: 18372\n",
      "MF_BPR: Epoch 348 of 1500. Elapsed time 1.17 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.51 seconds. BPR loss 5.34E+00. Sample per second: 15453\n",
      "MF_BPR: Epoch 349 of 1500. Elapsed time 1.17 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.25 seconds. BPR loss 5.36E+00. Sample per second: 16390\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.31 sec. Users per second: 1947\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.78 sec. Users per second: 1946\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2273676, PRECISION: 0.0789445, PRECISION_RECALL_MIN_DEN: 0.0999560, RECALL: 0.0650417, MAP: 0.0458899, MRR: 0.1874803, NDCG: 0.0640152, F1: 0.0713219, HIT_RATE: 0.7894450, ARHR: 0.2605253, NOVELTY: 0.0098238, AVERAGE_POPULARITY: 0.5193817, DIVERSITY_MEAN_INTER_LIST: 0.8785302, DIVERSITY_HERFINDAHL: 0.9878518, COVERAGE_ITEM: 0.9957869, COVERAGE_ITEM_CORRECT: 0.0578598, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4042760, DIVERSITY_GINI: 0.1618937, SHANNON_ENTROPY: 8.6916017, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 350 of 1500. Elapsed time 1.19 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.79 seconds. BPR loss 5.33E+00. Sample per second: 18366\n",
      "MF_BPR: Epoch 351 of 1500. Elapsed time 1.19 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.44 seconds. BPR loss 5.48E+00. Sample per second: 15670\n",
      "MF_BPR: Epoch 352 of 1500. Elapsed time 1.19 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.10 seconds. BPR loss 5.42E+00. Sample per second: 16972\n",
      "MF_BPR: Epoch 353 of 1500. Elapsed time 1.19 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.77 seconds. BPR loss 5.52E+00. Sample per second: 18457\n",
      "MF_BPR: Epoch 354 of 1500. Elapsed time 1.19 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.46 seconds. BPR loss 5.53E+00. Sample per second: 15599\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.40 sec. Users per second: 1941\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.71 sec. Users per second: 1950\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2277209, PRECISION: 0.0791068, PRECISION_RECALL_MIN_DEN: 0.1001299, RECALL: 0.0650959, MAP: 0.0459704, MRR: 0.1878754, NDCG: 0.0641518, F1: 0.0714207, HIT_RATE: 0.7910677, ARHR: 0.2610426, NOVELTY: 0.0098181, AVERAGE_POPULARITY: 0.5200054, DIVERSITY_MEAN_INTER_LIST: 0.8783357, DIVERSITY_HERFINDAHL: 0.9878323, COVERAGE_ITEM: 0.9957869, COVERAGE_ITEM_CORRECT: 0.0577661, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4048914, DIVERSITY_GINI: 0.1610969, SHANNON_ENTROPY: 8.6869989, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 355 of 1500. Elapsed time 1.20 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.91 seconds. BPR loss 5.61E+00. Sample per second: 17801\n",
      "MF_BPR: Epoch 356 of 1500. Elapsed time 1.20 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.60 seconds. BPR loss 5.54E+00. Sample per second: 15148\n",
      "MF_BPR: Epoch 357 of 1500. Elapsed time 1.20 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.33 seconds. BPR loss 5.60E+00. Sample per second: 16088\n",
      "MF_BPR: Epoch 358 of 1500. Elapsed time 1.20 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.00 seconds. BPR loss 5.71E+00. Sample per second: 17388\n",
      "MF_BPR: Epoch 359 of 1500. Elapsed time 1.20 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.63 seconds. BPR loss 5.69E+00. Sample per second: 19173\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.03 sec. Users per second: 1965\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.14 sec. Users per second: 1982\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2283087, PRECISION: 0.0792777, PRECISION_RECALL_MIN_DEN: 0.1002972, RECALL: 0.0651597, MAP: 0.0460711, MRR: 0.1882874, NDCG: 0.0643225, F1: 0.0715287, HIT_RATE: 0.7927766, ARHR: 0.2617200, NOVELTY: 0.0098123, AVERAGE_POPULARITY: 0.5208479, DIVERSITY_MEAN_INTER_LIST: 0.8779726, DIVERSITY_HERFINDAHL: 0.9877960, COVERAGE_ITEM: 0.9957869, COVERAGE_ITEM_CORRECT: 0.0573916, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4054638, DIVERSITY_GINI: 0.1603530, SHANNON_ENTROPY: 8.6820837, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 360 of 1500. Elapsed time 1.22 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.55 seconds. BPR loss 5.72E+00. Sample per second: 15291\n",
      "MF_BPR: Epoch 361 of 1500. Elapsed time 1.22 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.22 seconds. BPR loss 5.79E+00. Sample per second: 16507\n",
      "MF_BPR: Epoch 362 of 1500. Elapsed time 1.22 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.93 seconds. BPR loss 5.84E+00. Sample per second: 17738\n",
      "MF_BPR: Epoch 363 of 1500. Elapsed time 1.22 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.59 seconds. BPR loss 5.82E+00. Sample per second: 15158\n",
      "MF_BPR: Epoch 364 of 1500. Elapsed time 1.22 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.28 seconds. BPR loss 5.89E+00. Sample per second: 16277\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 61000 ( 87.60% ) in 30.19 sec. Users per second: 2021\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 34.54 sec. Users per second: 2016\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2285892, PRECISION: 0.0794543, PRECISION_RECALL_MIN_DEN: 0.1004413, RECALL: 0.0651817, MAP: 0.0461011, MRR: 0.1885067, NDCG: 0.0644230, F1: 0.0716138, HIT_RATE: 0.7945430, ARHR: 0.2620500, NOVELTY: 0.0098065, AVERAGE_POPULARITY: 0.5215734, DIVERSITY_MEAN_INTER_LIST: 0.8777531, DIVERSITY_HERFINDAHL: 0.9877741, COVERAGE_ITEM: 0.9955997, COVERAGE_ITEM_CORRECT: 0.0578598, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4063081, DIVERSITY_GINI: 0.1595835, SHANNON_ENTROPY: 8.6778529, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 365 of 1500. Elapsed time 1.23 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.77 seconds. BPR loss 5.90E+00. Sample per second: 14595\n",
      "MF_BPR: Epoch 366 of 1500. Elapsed time 1.23 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.54 seconds. BPR loss 5.97E+00. Sample per second: 15349\n",
      "MF_BPR: Epoch 367 of 1500. Elapsed time 1.23 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.33 seconds. BPR loss 6.03E+00. Sample per second: 16066\n",
      "MF_BPR: Epoch 368 of 1500. Elapsed time 1.23 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.00 seconds. BPR loss 5.99E+00. Sample per second: 17424\n",
      "MF_BPR: Epoch 369 of 1500. Elapsed time 1.23 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.69 seconds. BPR loss 6.02E+00. Sample per second: 18852\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 60000 ( 86.16% ) in 30.16 sec. Users per second: 1989\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 34.95 sec. Users per second: 1992\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2291896, PRECISION: 0.0795534, PRECISION_RECALL_MIN_DEN: 0.1004634, RECALL: 0.0651167, MAP: 0.0461680, MRR: 0.1887102, NDCG: 0.0645526, F1: 0.0716147, HIT_RATE: 0.7955339, ARHR: 0.2623954, NOVELTY: 0.0098016, AVERAGE_POPULARITY: 0.5220667, DIVERSITY_MEAN_INTER_LIST: 0.8776301, DIVERSITY_HERFINDAHL: 0.9877617, COVERAGE_ITEM: 0.9955060, COVERAGE_ITEM_CORRECT: 0.0575789, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4068233, DIVERSITY_GINI: 0.1588851, SHANNON_ENTROPY: 8.6750937, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 370 of 1500. Elapsed time 1.25 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.41 seconds. BPR loss 6.01E+00. Sample per second: 15803\n",
      "MF_BPR: Epoch 371 of 1500. Elapsed time 1.25 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.08 seconds. BPR loss 6.13E+00. Sample per second: 17049\n",
      "MF_BPR: Epoch 372 of 1500. Elapsed time 1.25 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.77 seconds. BPR loss 6.18E+00. Sample per second: 18473\n",
      "MF_BPR: Epoch 373 of 1500. Elapsed time 1.25 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.40 seconds. BPR loss 6.13E+00. Sample per second: 15838\n",
      "MF_BPR: Epoch 374 of 1500. Elapsed time 1.25 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.03 seconds. BPR loss 6.22E+00. Sample per second: 17295\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.01 sec. Users per second: 1966\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.37 sec. Users per second: 1969\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2295689, PRECISION: 0.0797042, PRECISION_RECALL_MIN_DEN: 0.1006277, RECALL: 0.0651764, MAP: 0.0462365, MRR: 0.1890378, NDCG: 0.0646894, F1: 0.0717119, HIT_RATE: 0.7970417, ARHR: 0.2628969, NOVELTY: 0.0097975, AVERAGE_POPULARITY: 0.5223504, DIVERSITY_MEAN_INTER_LIST: 0.8778020, DIVERSITY_HERFINDAHL: 0.9877789, COVERAGE_ITEM: 0.9954124, COVERAGE_ITEM_CORRECT: 0.0574853, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4078394, DIVERSITY_GINI: 0.1582489, SHANNON_ENTROPY: 8.6739156, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 375 of 1500. Elapsed time 1.26 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.20 seconds. BPR loss 6.15E+00. Sample per second: 16579\n",
      "MF_BPR: Epoch 376 of 1500. Elapsed time 1.26 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.92 seconds. BPR loss 6.23E+00. Sample per second: 17784\n",
      "MF_BPR: Epoch 377 of 1500. Elapsed time 1.26 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.67 seconds. BPR loss 6.25E+00. Sample per second: 14903\n",
      "MF_BPR: Epoch 378 of 1500. Elapsed time 1.26 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.44 seconds. BPR loss 6.31E+00. Sample per second: 15696\n",
      "MF_BPR: Epoch 379 of 1500. Elapsed time 1.26 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.16 seconds. BPR loss 6.36E+00. Sample per second: 16746\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.21 sec. Users per second: 1953\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.53 sec. Users per second: 1960\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2295839, PRECISION: 0.0798679, PRECISION_RECALL_MIN_DEN: 0.1008159, RECALL: 0.0652704, MAP: 0.0462842, MRR: 0.1891553, NDCG: 0.0647933, F1: 0.0718351, HIT_RATE: 0.7986788, ARHR: 0.2631644, NOVELTY: 0.0097940, AVERAGE_POPULARITY: 0.5225201, DIVERSITY_MEAN_INTER_LIST: 0.8779240, DIVERSITY_HERFINDAHL: 0.9877911, COVERAGE_ITEM: 0.9955060, COVERAGE_ITEM_CORRECT: 0.0574853, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4085692, DIVERSITY_GINI: 0.1576532, SHANNON_ENTROPY: 8.6732984, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 380 of 1500. Elapsed time 1.28 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.45 seconds. BPR loss 6.40E+00. Sample per second: 15635\n",
      "MF_BPR: Epoch 381 of 1500. Elapsed time 1.28 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.08 seconds. BPR loss 6.47E+00. Sample per second: 17060\n",
      "MF_BPR: Epoch 382 of 1500. Elapsed time 1.28 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.72 seconds. BPR loss 6.52E+00. Sample per second: 18697\n",
      "MF_BPR: Epoch 383 of 1500. Elapsed time 1.28 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.41 seconds. BPR loss 6.48E+00. Sample per second: 15789\n",
      "MF_BPR: Epoch 384 of 1500. Elapsed time 1.28 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.14 seconds. BPR loss 6.61E+00. Sample per second: 16813\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 58000 ( 83.29% ) in 30.03 sec. Users per second: 1932\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 36.01 sec. Users per second: 1934\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2296919, PRECISION: 0.0799785, PRECISION_RECALL_MIN_DEN: 0.1008713, RECALL: 0.0652580, MAP: 0.0463399, MRR: 0.1893558, NDCG: 0.0648994, F1: 0.0718722, HIT_RATE: 0.7997846, ARHR: 0.2635703, NOVELTY: 0.0097911, AVERAGE_POPULARITY: 0.5227099, DIVERSITY_MEAN_INTER_LIST: 0.8780671, DIVERSITY_HERFINDAHL: 0.9878054, COVERAGE_ITEM: 0.9953188, COVERAGE_ITEM_CORRECT: 0.0573916, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4086837, DIVERSITY_GINI: 0.1571278, SHANNON_ENTROPY: 8.6736448, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 385 of 1500. Elapsed time 1.29 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.96 seconds. BPR loss 6.63E+00. Sample per second: 17597\n",
      "MF_BPR: Epoch 386 of 1500. Elapsed time 1.29 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.67 seconds. BPR loss 6.60E+00. Sample per second: 14901\n",
      "MF_BPR: Epoch 387 of 1500. Elapsed time 1.29 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.34 seconds. BPR loss 6.61E+00. Sample per second: 16037\n",
      "MF_BPR: Epoch 388 of 1500. Elapsed time 1.29 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.06 seconds. BPR loss 6.67E+00. Sample per second: 17132\n",
      "MF_BPR: Epoch 389 of 1500. Elapsed time 1.29 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.72 seconds. BPR loss 6.78E+00. Sample per second: 18709\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.13 sec. Users per second: 1958\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.44 sec. Users per second: 1965\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2296815, PRECISION: 0.0801134, PRECISION_RECALL_MIN_DEN: 0.1010355, RECALL: 0.0653478, MAP: 0.0463942, MRR: 0.1894308, NDCG: 0.0649737, F1: 0.0719812, HIT_RATE: 0.8011345, ARHR: 0.2638289, NOVELTY: 0.0097884, AVERAGE_POPULARITY: 0.5227855, DIVERSITY_MEAN_INTER_LIST: 0.8782505, DIVERSITY_HERFINDAHL: 0.9878238, COVERAGE_ITEM: 0.9951315, COVERAGE_ITEM_CORRECT: 0.0571108, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4090701, DIVERSITY_GINI: 0.1565870, SHANNON_ENTROPY: 8.6735676, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 390 of 1500. Elapsed time 1.31 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.96 seconds. BPR loss 6.74E+00. Sample per second: 17584\n",
      "MF_BPR: Epoch 391 of 1500. Elapsed time 1.31 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.65 seconds. BPR loss 6.80E+00. Sample per second: 14989\n",
      "MF_BPR: Epoch 392 of 1500. Elapsed time 1.31 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.43 seconds. BPR loss 6.80E+00. Sample per second: 15725\n",
      "MF_BPR: Epoch 393 of 1500. Elapsed time 1.31 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.21 seconds. BPR loss 6.85E+00. Sample per second: 16529\n",
      "MF_BPR: Epoch 394 of 1500. Elapsed time 1.31 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.91 seconds. BPR loss 6.87E+00. Sample per second: 17798\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 57000 ( 81.86% ) in 30.08 sec. Users per second: 1895\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 36.25 sec. Users per second: 1921\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2297345, PRECISION: 0.0802671, PRECISION_RECALL_MIN_DEN: 0.1012191, RECALL: 0.0654479, MAP: 0.0464227, MRR: 0.1894058, NDCG: 0.0651137, F1: 0.0721040, HIT_RATE: 0.8026711, ARHR: 0.2639621, NOVELTY: 0.0097861, AVERAGE_POPULARITY: 0.5229169, DIVERSITY_MEAN_INTER_LIST: 0.8783542, DIVERSITY_HERFINDAHL: 0.9878342, COVERAGE_ITEM: 0.9951315, COVERAGE_ITEM_CORRECT: 0.0575789, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4095853, DIVERSITY_GINI: 0.1560669, SHANNON_ENTROPY: 8.6738532, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 395 of 1500. Elapsed time 1.32 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.91 seconds. BPR loss 6.87E+00. Sample per second: 17795\n",
      "MF_BPR: Epoch 396 of 1500. Elapsed time 1.32 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.54 seconds. BPR loss 6.88E+00. Sample per second: 15331\n",
      "MF_BPR: Epoch 397 of 1500. Elapsed time 1.32 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.17 seconds. BPR loss 6.91E+00. Sample per second: 16705\n",
      "MF_BPR: Epoch 398 of 1500. Elapsed time 1.32 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.80 seconds. BPR loss 6.98E+00. Sample per second: 18322\n",
      "MF_BPR: Epoch 399 of 1500. Elapsed time 1.32 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.43 seconds. BPR loss 7.01E+00. Sample per second: 15724\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 60000 ( 86.16% ) in 30.36 sec. Users per second: 1976\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.10 sec. Users per second: 1984\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2297274, PRECISION: 0.0804021, PRECISION_RECALL_MIN_DEN: 0.1013873, RECALL: 0.0655242, MAP: 0.0465142, MRR: 0.1895877, NDCG: 0.0652246, F1: 0.0722047, HIT_RATE: 0.8040210, ARHR: 0.2643537, NOVELTY: 0.0097834, AVERAGE_POPULARITY: 0.5230174, DIVERSITY_MEAN_INTER_LIST: 0.8784225, DIVERSITY_HERFINDAHL: 0.9878410, COVERAGE_ITEM: 0.9951315, COVERAGE_ITEM_CORRECT: 0.0573916, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4096998, DIVERSITY_GINI: 0.1555956, SHANNON_ENTROPY: 8.6737546, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 400 of 1500. Elapsed time 1.34 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.23 seconds. BPR loss 7.08E+00. Sample per second: 16471\n",
      "MF_BPR: Epoch 401 of 1500. Elapsed time 1.34 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.01 seconds. BPR loss 7.03E+00. Sample per second: 17373\n",
      "MF_BPR: Epoch 402 of 1500. Elapsed time 1.34 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.66 seconds. BPR loss 7.14E+00. Sample per second: 19016\n",
      "MF_BPR: Epoch 403 of 1500. Elapsed time 1.34 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.30 seconds. BPR loss 7.20E+00. Sample per second: 16201\n",
      "MF_BPR: Epoch 404 of 1500. Elapsed time 1.34 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.99 seconds. BPR loss 7.21E+00. Sample per second: 17438\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 60000 ( 86.16% ) in 30.38 sec. Users per second: 1975\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.20 sec. Users per second: 1978\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2297480, PRECISION: 0.0805098, PRECISION_RECALL_MIN_DEN: 0.1014747, RECALL: 0.0655446, MAP: 0.0465484, MRR: 0.1897707, NDCG: 0.0653082, F1: 0.0722605, HIT_RATE: 0.8050980, ARHR: 0.2646642, NOVELTY: 0.0097810, AVERAGE_POPULARITY: 0.5230248, DIVERSITY_MEAN_INTER_LIST: 0.8786124, DIVERSITY_HERFINDAHL: 0.9878600, COVERAGE_ITEM: 0.9949443, COVERAGE_ITEM_CORRECT: 0.0573916, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4099717, DIVERSITY_GINI: 0.1550302, SHANNON_ENTROPY: 8.6744328, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 405 of 1500. Elapsed time 1.35 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.92 seconds. BPR loss 7.14E+00. Sample per second: 17746\n",
      "MF_BPR: Epoch 406 of 1500. Elapsed time 1.35 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.56 seconds. BPR loss 7.22E+00. Sample per second: 15271\n",
      "MF_BPR: Epoch 407 of 1500. Elapsed time 1.35 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.24 seconds. BPR loss 7.27E+00. Sample per second: 16442\n",
      "MF_BPR: Epoch 408 of 1500. Elapsed time 1.35 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.02 seconds. BPR loss 7.30E+00. Sample per second: 17328\n",
      "MF_BPR: Epoch 409 of 1500. Elapsed time 1.35 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.75 seconds. BPR loss 7.35E+00. Sample per second: 18566\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.50 sec. Users per second: 1836\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.52 sec. Users per second: 1856\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2298811, PRECISION: 0.0806893, PRECISION_RECALL_MIN_DEN: 0.1016803, RECALL: 0.0656460, MAP: 0.0466523, MRR: 0.1899211, NDCG: 0.0654164, F1: 0.0723944, HIT_RATE: 0.8068931, ARHR: 0.2650424, NOVELTY: 0.0097792, AVERAGE_POPULARITY: 0.5228727, DIVERSITY_MEAN_INTER_LIST: 0.8790566, DIVERSITY_HERFINDAHL: 0.9879044, COVERAGE_ITEM: 0.9948507, COVERAGE_ITEM_CORRECT: 0.0577661, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4106443, DIVERSITY_GINI: 0.1545996, SHANNON_ENTROPY: 8.6772613, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 410 of 1500. Elapsed time 1.37 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.09 seconds. BPR loss 7.35E+00. Sample per second: 17010\n",
      "MF_BPR: Epoch 411 of 1500. Elapsed time 1.37 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.95 seconds. BPR loss 7.46E+00. Sample per second: 17634\n",
      "MF_BPR: Epoch 412 of 1500. Elapsed time 1.37 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.79 seconds. BPR loss 7.41E+00. Sample per second: 14538\n",
      "MF_BPR: Epoch 413 of 1500. Elapsed time 1.37 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.58 seconds. BPR loss 7.52E+00. Sample per second: 15207\n",
      "MF_BPR: Epoch 414 of 1500. Elapsed time 1.37 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.34 seconds. BPR loss 7.47E+00. Sample per second: 16035\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.12 sec. Users per second: 1859\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.24 sec. Users per second: 1870\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2299041, PRECISION: 0.0808315, PRECISION_RECALL_MIN_DEN: 0.1018616, RECALL: 0.0657662, MAP: 0.0466971, MRR: 0.1900207, NDCG: 0.0655439, F1: 0.0725248, HIT_RATE: 0.8083148, ARHR: 0.2653061, NOVELTY: 0.0097771, AVERAGE_POPULARITY: 0.5228141, DIVERSITY_MEAN_INTER_LIST: 0.8794006, DIVERSITY_HERFINDAHL: 0.9879388, COVERAGE_ITEM: 0.9949443, COVERAGE_ITEM_CORRECT: 0.0577661, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4111165, DIVERSITY_GINI: 0.1541307, SHANNON_ENTROPY: 8.6790782, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 415 of 1500. Elapsed time 1.38 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.47 seconds. BPR loss 7.50E+00. Sample per second: 15587\n",
      "MF_BPR: Epoch 416 of 1500. Elapsed time 1.38 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.54 seconds. BPR loss 7.55E+00. Sample per second: 15335\n",
      "MF_BPR: Epoch 417 of 1500. Elapsed time 1.38 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.49 seconds. BPR loss 7.61E+00. Sample per second: 15515\n",
      "MF_BPR: Epoch 418 of 1500. Elapsed time 1.38 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.62 seconds. BPR loss 7.53E+00. Sample per second: 15086\n",
      "MF_BPR: Epoch 419 of 1500. Elapsed time 1.39 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.72 seconds. BPR loss 7.60E+00. Sample per second: 14743\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.45 sec. Users per second: 1839\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.30 sec. Users per second: 1867\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2300540, PRECISION: 0.0808961, PRECISION_RECALL_MIN_DEN: 0.1018942, RECALL: 0.0657657, MAP: 0.0467448, MRR: 0.1901888, NDCG: 0.0656246, F1: 0.0725504, HIT_RATE: 0.8089610, ARHR: 0.2655485, NOVELTY: 0.0097755, AVERAGE_POPULARITY: 0.5225380, DIVERSITY_MEAN_INTER_LIST: 0.8797856, DIVERSITY_HERFINDAHL: 0.9879773, COVERAGE_ITEM: 0.9949443, COVERAGE_ITEM_CORRECT: 0.0576725, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4112453, DIVERSITY_GINI: 0.1536359, SHANNON_ENTROPY: 8.6816861, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 420 of 1500. Elapsed time 1.40 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.95 seconds. BPR loss 7.72E+00. Sample per second: 17616\n",
      "MF_BPR: Epoch 421 of 1500. Elapsed time 1.40 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.94 seconds. BPR loss 7.65E+00. Sample per second: 14093\n",
      "MF_BPR: Epoch 422 of 1500. Elapsed time 1.40 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.07 seconds. BPR loss 7.69E+00. Sample per second: 13729\n",
      "MF_BPR: Epoch 423 of 1500. Elapsed time 1.40 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.10 seconds. BPR loss 7.69E+00. Sample per second: 16990\n",
      "MF_BPR: Epoch 424 of 1500. Elapsed time 1.40 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.11 seconds. BPR loss 7.80E+00. Sample per second: 16945\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.20 sec. Users per second: 1854\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.03 sec. Users per second: 1881\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2299238, PRECISION: 0.0810383, PRECISION_RECALL_MIN_DEN: 0.1020486, RECALL: 0.0658289, MAP: 0.0467812, MRR: 0.1901782, NDCG: 0.0657125, F1: 0.0726461, HIT_RATE: 0.8103827, ARHR: 0.2656730, NOVELTY: 0.0097739, AVERAGE_POPULARITY: 0.5223587, DIVERSITY_MEAN_INTER_LIST: 0.8801862, DIVERSITY_HERFINDAHL: 0.9880174, COVERAGE_ITEM: 0.9950379, COVERAGE_ITEM_CORRECT: 0.0581406, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4116746, DIVERSITY_GINI: 0.1531375, SHANNON_ENTROPY: 8.6840119, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 425 of 1500. Elapsed time 1.41 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.12 seconds. BPR loss 7.74E+00. Sample per second: 16902\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 426 of 1500. Elapsed time 1.41 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.93 seconds. BPR loss 7.82E+00. Sample per second: 17700\n",
      "MF_BPR: Epoch 427 of 1500. Elapsed time 1.42 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.73 seconds. BPR loss 7.88E+00. Sample per second: 14730\n",
      "MF_BPR: Epoch 428 of 1500. Elapsed time 1.42 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.56 seconds. BPR loss 7.84E+00. Sample per second: 15263\n",
      "MF_BPR: Epoch 429 of 1500. Elapsed time 1.42 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.31 seconds. BPR loss 7.97E+00. Sample per second: 16175\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 60000 ( 86.16% ) in 30.25 sec. Users per second: 1983\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.41 sec. Users per second: 1967\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2300928, PRECISION: 0.0811173, PRECISION_RECALL_MIN_DEN: 0.1021175, RECALL: 0.0658608, MAP: 0.0467774, MRR: 0.1901984, NDCG: 0.0657837, F1: 0.0726972, HIT_RATE: 0.8111725, ARHR: 0.2657750, NOVELTY: 0.0097728, AVERAGE_POPULARITY: 0.5222021, DIVERSITY_MEAN_INTER_LIST: 0.8804769, DIVERSITY_HERFINDAHL: 0.9880464, COVERAGE_ITEM: 0.9952252, COVERAGE_ITEM_CORRECT: 0.0579534, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4120467, DIVERSITY_GINI: 0.1527141, SHANNON_ENTROPY: 8.6862998, \n",
      "\n",
      "MF_BPR: Epoch 430 of 1500. Elapsed time 1.43 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.98 seconds. BPR loss 7.92E+00. Sample per second: 13981\n",
      "MF_BPR: Epoch 431 of 1500. Elapsed time 1.43 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.05 seconds. BPR loss 8.02E+00. Sample per second: 13802\n",
      "MF_BPR: Epoch 432 of 1500. Elapsed time 1.43 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.92 seconds. BPR loss 8.02E+00. Sample per second: 17742\n",
      "MF_BPR: Epoch 433 of 1500. Elapsed time 1.43 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.73 seconds. BPR loss 8.04E+00. Sample per second: 14720\n",
      "MF_BPR: Epoch 434 of 1500. Elapsed time 1.43 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.42 seconds. BPR loss 7.99E+00. Sample per second: 15761\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.32 sec. Users per second: 1847\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.60 sec. Users per second: 1852\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2299016, PRECISION: 0.0812594, PRECISION_RECALL_MIN_DEN: 0.1022658, RECALL: 0.0658954, MAP: 0.0468372, MRR: 0.1901274, NDCG: 0.0658436, F1: 0.0727754, HIT_RATE: 0.8125942, ARHR: 0.2658955, NOVELTY: 0.0097715, AVERAGE_POPULARITY: 0.5218589, DIVERSITY_MEAN_INTER_LIST: 0.8810313, DIVERSITY_HERFINDAHL: 0.9881019, COVERAGE_ITEM: 0.9950379, COVERAGE_ITEM_CORRECT: 0.0578598, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4122471, DIVERSITY_GINI: 0.1523014, SHANNON_ENTROPY: 8.6888999, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 435 of 1500. Elapsed time 1.44 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.28 seconds. BPR loss 8.07E+00. Sample per second: 16265\n",
      "MF_BPR: Epoch 436 of 1500. Elapsed time 1.45 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.32 seconds. BPR loss 8.13E+00. Sample per second: 16118\n",
      "MF_BPR: Epoch 437 of 1500. Elapsed time 1.45 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.33 seconds. BPR loss 8.17E+00. Sample per second: 16070\n",
      "MF_BPR: Epoch 438 of 1500. Elapsed time 1.45 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.32 seconds. BPR loss 8.20E+00. Sample per second: 16101\n",
      "MF_BPR: Epoch 439 of 1500. Elapsed time 1.45 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.43 seconds. BPR loss 8.16E+00. Sample per second: 15710\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.48 sec. Users per second: 1804\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.32 sec. Users per second: 1817\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2300948, PRECISION: 0.0813743, PRECISION_RECALL_MIN_DEN: 0.1023698, RECALL: 0.0659200, MAP: 0.0469179, MRR: 0.1903611, NDCG: 0.0659151, F1: 0.0728364, HIT_RATE: 0.8137431, ARHR: 0.2662861, NOVELTY: 0.0097705, AVERAGE_POPULARITY: 0.5215509, DIVERSITY_MEAN_INTER_LIST: 0.8816319, DIVERSITY_HERFINDAHL: 0.9881619, COVERAGE_ITEM: 0.9952252, COVERAGE_ITEM_CORRECT: 0.0582342, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4124188, DIVERSITY_GINI: 0.1519377, SHANNON_ENTROPY: 8.6930396, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 440 of 1500. Elapsed time 1.46 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.88 seconds. BPR loss 8.24E+00. Sample per second: 14281\n",
      "MF_BPR: Epoch 441 of 1500. Elapsed time 1.46 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.95 seconds. BPR loss 8.20E+00. Sample per second: 14075\n",
      "MF_BPR: Epoch 442 of 1500. Elapsed time 1.46 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.96 seconds. BPR loss 8.22E+00. Sample per second: 14033\n",
      "MF_BPR: Epoch 443 of 1500. Elapsed time 1.46 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.02 seconds. BPR loss 8.31E+00. Sample per second: 13874\n",
      "MF_BPR: Epoch 444 of 1500. Elapsed time 1.47 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.85 seconds. BPR loss 8.32E+00. Sample per second: 18105\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.19 sec. Users per second: 1789\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.53 sec. Users per second: 1807\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2302226, PRECISION: 0.0814533, PRECISION_RECALL_MIN_DEN: 0.1024714, RECALL: 0.0659599, MAP: 0.0469751, MRR: 0.1906133, NDCG: 0.0660092, F1: 0.0728924, HIT_RATE: 0.8145329, ARHR: 0.2666252, NOVELTY: 0.0097693, AVERAGE_POPULARITY: 0.5214010, DIVERSITY_MEAN_INTER_LIST: 0.8818088, DIVERSITY_HERFINDAHL: 0.9881796, COVERAGE_ITEM: 0.9951315, COVERAGE_ITEM_CORRECT: 0.0576725, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4124617, DIVERSITY_GINI: 0.1515840, SHANNON_ENTROPY: 8.6950412, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 445 of 1500. Elapsed time 1.48 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.50 seconds. BPR loss 8.44E+00. Sample per second: 15458\n",
      "MF_BPR: Epoch 446 of 1500. Elapsed time 1.48 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.52 seconds. BPR loss 8.39E+00. Sample per second: 15422\n",
      "MF_BPR: Epoch 447 of 1500. Elapsed time 1.48 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.54 seconds. BPR loss 8.45E+00. Sample per second: 15337\n",
      "MF_BPR: Epoch 448 of 1500. Elapsed time 1.48 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.48 seconds. BPR loss 8.48E+00. Sample per second: 15542\n",
      "MF_BPR: Epoch 449 of 1500. Elapsed time 1.48 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.40 seconds. BPR loss 8.47E+00. Sample per second: 15810\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.50 sec. Users per second: 1836\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.27 sec. Users per second: 1819\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2302862, PRECISION: 0.0815093, PRECISION_RECALL_MIN_DEN: 0.1024666, RECALL: 0.0659030, MAP: 0.0470064, MRR: 0.1907019, NDCG: 0.0661113, F1: 0.0728800, HIT_RATE: 0.8150930, ARHR: 0.2668433, NOVELTY: 0.0097680, AVERAGE_POPULARITY: 0.5212551, DIVERSITY_MEAN_INTER_LIST: 0.8819741, DIVERSITY_HERFINDAHL: 0.9881961, COVERAGE_ITEM: 0.9950379, COVERAGE_ITEM_CORRECT: 0.0577661, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4127479, DIVERSITY_GINI: 0.1510955, SHANNON_ENTROPY: 8.6962879, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 450 of 1500. Elapsed time 1.49 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.78 seconds. BPR loss 8.46E+00. Sample per second: 14577\n",
      "MF_BPR: Epoch 451 of 1500. Elapsed time 1.49 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.72 seconds. BPR loss 8.55E+00. Sample per second: 14753\n",
      "MF_BPR: Epoch 452 of 1500. Elapsed time 1.50 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.61 seconds. BPR loss 8.50E+00. Sample per second: 15090\n",
      "MF_BPR: Epoch 453 of 1500. Elapsed time 1.50 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.52 seconds. BPR loss 8.64E+00. Sample per second: 15418\n",
      "MF_BPR: Epoch 454 of 1500. Elapsed time 1.50 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.46 seconds. BPR loss 8.65E+00. Sample per second: 15620\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.23 sec. Users per second: 1753\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.65 sec. Users per second: 1756\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2300766, PRECISION: 0.0816429, PRECISION_RECALL_MIN_DEN: 0.1026128, RECALL: 0.0659693, MAP: 0.0470423, MRR: 0.1907426, NDCG: 0.0661666, F1: 0.0729740, HIT_RATE: 0.8164285, ARHR: 0.2669595, NOVELTY: 0.0097670, AVERAGE_POPULARITY: 0.5208979, DIVERSITY_MEAN_INTER_LIST: 0.8824220, DIVERSITY_HERFINDAHL: 0.9882409, COVERAGE_ITEM: 0.9949443, COVERAGE_ITEM_CORRECT: 0.0582342, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4131916, DIVERSITY_GINI: 0.1507229, SHANNON_ENTROPY: 8.6996425, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 455 of 1500. Elapsed time 1.51 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.14 seconds. BPR loss 8.65E+00. Sample per second: 16805\n",
      "MF_BPR: Epoch 456 of 1500. Elapsed time 1.51 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.18 seconds. BPR loss 8.68E+00. Sample per second: 16643\n",
      "MF_BPR: Epoch 457 of 1500. Elapsed time 1.51 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.19 seconds. BPR loss 8.65E+00. Sample per second: 16617\n",
      "MF_BPR: Epoch 458 of 1500. Elapsed time 1.51 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.16 seconds. BPR loss 8.73E+00. Sample per second: 16730\n",
      "MF_BPR: Epoch 459 of 1500. Elapsed time 1.51 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.13 seconds. BPR loss 8.74E+00. Sample per second: 16859\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.22 sec. Users per second: 1820\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.84 sec. Users per second: 1840\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2301509, PRECISION: 0.0817491, PRECISION_RECALL_MIN_DEN: 0.1027417, RECALL: 0.0660640, MAP: 0.0471185, MRR: 0.1907661, NDCG: 0.0662088, F1: 0.0730743, HIT_RATE: 0.8174912, ARHR: 0.2671637, NOVELTY: 0.0097666, AVERAGE_POPULARITY: 0.5205331, DIVERSITY_MEAN_INTER_LIST: 0.8830077, DIVERSITY_HERFINDAHL: 0.9882995, COVERAGE_ITEM: 0.9950379, COVERAGE_ITEM_CORRECT: 0.0579534, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4135493, DIVERSITY_GINI: 0.1503484, SHANNON_ENTROPY: 8.7033168, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 460 of 1500. Elapsed time 1.53 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.98 seconds. BPR loss 8.77E+00. Sample per second: 17481\n",
      "MF_BPR: Epoch 461 of 1500. Elapsed time 1.53 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.89 seconds. BPR loss 8.88E+00. Sample per second: 14240\n",
      "MF_BPR: Epoch 462 of 1500. Elapsed time 1.53 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.81 seconds. BPR loss 8.89E+00. Sample per second: 14468\n",
      "MF_BPR: Epoch 463 of 1500. Elapsed time 1.53 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.73 seconds. BPR loss 8.92E+00. Sample per second: 14729\n",
      "MF_BPR: Epoch 464 of 1500. Elapsed time 1.53 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.59 seconds. BPR loss 8.95E+00. Sample per second: 15165\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.43 sec. Users per second: 1840\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.70 sec. Users per second: 1847\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2302431, PRECISION: 0.0817577, PRECISION_RECALL_MIN_DEN: 0.1027495, RECALL: 0.0660789, MAP: 0.0471031, MRR: 0.1905993, NDCG: 0.0661995, F1: 0.0730869, HIT_RATE: 0.8175774, ARHR: 0.2670061, NOVELTY: 0.0097658, AVERAGE_POPULARITY: 0.5201509, DIVERSITY_MEAN_INTER_LIST: 0.8833512, DIVERSITY_HERFINDAHL: 0.9883339, COVERAGE_ITEM: 0.9949443, COVERAGE_ITEM_CORRECT: 0.0585151, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4134778, DIVERSITY_GINI: 0.1499383, SHANNON_ENTROPY: 8.7063852, \n",
      "\n",
      "MF_BPR: Epoch 465 of 1500. Elapsed time 1.54 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.02 seconds. BPR loss 8.96E+00. Sample per second: 17303\n",
      "MF_BPR: Epoch 466 of 1500. Elapsed time 1.54 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 3.70 seconds. BPR loss 9.03E+00. Sample per second: 18828\n",
      "MF_BPR: Epoch 467 of 1500. Elapsed time 1.54 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.35 seconds. BPR loss 8.97E+00. Sample per second: 15995\n",
      "MF_BPR: Epoch 468 of 1500. Elapsed time 1.54 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.41 seconds. BPR loss 9.02E+00. Sample per second: 15804\n",
      "MF_BPR: Epoch 469 of 1500. Elapsed time 1.55 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.66 seconds. BPR loss 9.07E+00. Sample per second: 14930\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.51 sec. Users per second: 1737\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.11 sec. Users per second: 1736\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2301764, PRECISION: 0.0818755, PRECISION_RECALL_MIN_DEN: 0.1028994, RECALL: 0.0661730, MAP: 0.0471799, MRR: 0.1908211, NDCG: 0.0662851, F1: 0.0731915, HIT_RATE: 0.8187549, ARHR: 0.2673472, NOVELTY: 0.0097654, AVERAGE_POPULARITY: 0.5195916, DIVERSITY_MEAN_INTER_LIST: 0.8840013, DIVERSITY_HERFINDAHL: 0.9883989, COVERAGE_ITEM: 0.9949443, COVERAGE_ITEM_CORRECT: 0.0580470, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4136066, DIVERSITY_GINI: 0.1496689, SHANNON_ENTROPY: 8.7110604, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 470 of 1500. Elapsed time 1.56 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.01 seconds. BPR loss 9.13E+00. Sample per second: 13912\n",
      "MF_BPR: Epoch 471 of 1500. Elapsed time 1.56 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.20 seconds. BPR loss 9.10E+00. Sample per second: 16571\n",
      "MF_BPR: Epoch 472 of 1500. Elapsed time 1.56 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.38 seconds. BPR loss 9.08E+00. Sample per second: 15912\n",
      "MF_BPR: Epoch 473 of 1500. Elapsed time 1.56 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.70 seconds. BPR loss 9.19E+00. Sample per second: 14804\n",
      "MF_BPR: Epoch 474 of 1500. Elapsed time 1.56 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.07 seconds. BPR loss 9.15E+00. Sample per second: 13741\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.27 sec. Users per second: 1685\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.14 sec. Users per second: 1692\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2302770, PRECISION: 0.0819631, PRECISION_RECALL_MIN_DEN: 0.1030270, RECALL: 0.0662680, MAP: 0.0472115, MRR: 0.1908699, NDCG: 0.0663400, F1: 0.0732846, HIT_RATE: 0.8196309, ARHR: 0.2674701, NOVELTY: 0.0097641, AVERAGE_POPULARITY: 0.5192819, DIVERSITY_MEAN_INTER_LIST: 0.8843033, DIVERSITY_HERFINDAHL: 0.9884291, COVERAGE_ITEM: 0.9948507, COVERAGE_ITEM_CORRECT: 0.0580470, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4140645, DIVERSITY_GINI: 0.1492173, SHANNON_ENTROPY: 8.7131055, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 475 of 1500. Elapsed time 1.58 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.44 seconds. BPR loss 9.26E+00. Sample per second: 15666\n",
      "MF_BPR: Epoch 476 of 1500. Elapsed time 1.58 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.57 seconds. BPR loss 9.19E+00. Sample per second: 15245\n",
      "MF_BPR: Epoch 477 of 1500. Elapsed time 1.58 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.69 seconds. BPR loss 9.19E+00. Sample per second: 14859\n",
      "MF_BPR: Epoch 478 of 1500. Elapsed time 1.58 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.78 seconds. BPR loss 9.21E+00. Sample per second: 14555\n",
      "MF_BPR: Epoch 479 of 1500. Elapsed time 1.58 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.81 seconds. BPR loss 9.30E+00. Sample per second: 14487\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.50 sec. Users per second: 1705\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.45 sec. Users per second: 1721\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2301635, PRECISION: 0.0820263, PRECISION_RECALL_MIN_DEN: 0.1031053, RECALL: 0.0663123, MAP: 0.0472832, MRR: 0.1910647, NDCG: 0.0664172, F1: 0.0733370, HIT_RATE: 0.8202628, ARHR: 0.2678024, NOVELTY: 0.0097634, AVERAGE_POPULARITY: 0.5189729, DIVERSITY_MEAN_INTER_LIST: 0.8846067, DIVERSITY_HERFINDAHL: 0.9884594, COVERAGE_ITEM: 0.9950379, COVERAGE_ITEM_CORRECT: 0.0580470, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4138212, DIVERSITY_GINI: 0.1489182, SHANNON_ENTROPY: 8.7159602, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 480 of 1500. Elapsed time 1.59 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.49 seconds. BPR loss 9.41E+00. Sample per second: 15491\n",
      "MF_BPR: Epoch 481 of 1500. Elapsed time 1.59 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.65 seconds. BPR loss 9.32E+00. Sample per second: 14973\n",
      "MF_BPR: Epoch 482 of 1500. Elapsed time 1.59 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.86 seconds. BPR loss 9.34E+00. Sample per second: 14330\n",
      "MF_BPR: Epoch 483 of 1500. Elapsed time 1.60 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.91 seconds. BPR loss 9.49E+00. Sample per second: 14190\n",
      "MF_BPR: Epoch 484 of 1500. Elapsed time 1.60 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.98 seconds. BPR loss 9.47E+00. Sample per second: 13987\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.29 sec. Users per second: 1684\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.32 sec. Users per second: 1685\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2302082, PRECISION: 0.0820435, PRECISION_RECALL_MIN_DEN: 0.1031394, RECALL: 0.0663583, MAP: 0.0472732, MRR: 0.1908210, NDCG: 0.0664545, F1: 0.0733720, HIT_RATE: 0.8204351, ARHR: 0.2676124, NOVELTY: 0.0097634, AVERAGE_POPULARITY: 0.5185846, DIVERSITY_MEAN_INTER_LIST: 0.8850592, DIVERSITY_HERFINDAHL: 0.9885046, COVERAGE_ITEM: 0.9948507, COVERAGE_ITEM_CORRECT: 0.0579534, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4140216, DIVERSITY_GINI: 0.1485366, SHANNON_ENTROPY: 8.7197199, \n",
      "\n",
      "MF_BPR: Epoch 485 of 1500. Elapsed time 1.61 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.55 seconds. BPR loss 9.46E+00. Sample per second: 15321\n",
      "MF_BPR: Epoch 486 of 1500. Elapsed time 1.61 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.66 seconds. BPR loss 9.48E+00. Sample per second: 14948\n",
      "MF_BPR: Epoch 487 of 1500. Elapsed time 1.61 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.81 seconds. BPR loss 9.52E+00. Sample per second: 14479\n",
      "MF_BPR: Epoch 488 of 1500. Elapsed time 1.61 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.96 seconds. BPR loss 9.54E+00. Sample per second: 14036\n",
      "MF_BPR: Epoch 489 of 1500. Elapsed time 1.61 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.04 seconds. BPR loss 9.63E+00. Sample per second: 13804\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.17 sec. Users per second: 1691\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.83 sec. Users per second: 1706\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2302147, PRECISION: 0.0821124, PRECISION_RECALL_MIN_DEN: 0.1032670, RECALL: 0.0664447, MAP: 0.0473395, MRR: 0.1909478, NDCG: 0.0665402, F1: 0.0734524, HIT_RATE: 0.8211244, ARHR: 0.2677561, NOVELTY: 0.0097636, AVERAGE_POPULARITY: 0.5180674, DIVERSITY_MEAN_INTER_LIST: 0.8855206, DIVERSITY_HERFINDAHL: 0.9885508, COVERAGE_ITEM: 0.9944762, COVERAGE_ITEM_CORRECT: 0.0581406, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4142935, DIVERSITY_GINI: 0.1481480, SHANNON_ENTROPY: 8.7235074, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 490 of 1500. Elapsed time 1.63 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.06 seconds. BPR loss 9.61E+00. Sample per second: 13769\n",
      "MF_BPR: Epoch 491 of 1500. Elapsed time 1.63 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.12 seconds. BPR loss 9.59E+00. Sample per second: 16903\n",
      "MF_BPR: Epoch 492 of 1500. Elapsed time 1.63 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.15 seconds. BPR loss 9.65E+00. Sample per second: 16797\n",
      "MF_BPR: Epoch 493 of 1500. Elapsed time 1.63 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.23 seconds. BPR loss 9.67E+00. Sample per second: 16465\n",
      "MF_BPR: Epoch 494 of 1500. Elapsed time 1.63 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 9.65E+00. Sample per second: 15930\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.34 sec. Users per second: 1681\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.33 sec. Users per second: 1685\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2302502, PRECISION: 0.0821598, PRECISION_RECALL_MIN_DEN: 0.1033274, RECALL: 0.0664972, MAP: 0.0473357, MRR: 0.1908781, NDCG: 0.0665727, F1: 0.0735034, HIT_RATE: 0.8215983, ARHR: 0.2677510, NOVELTY: 0.0097630, AVERAGE_POPULARITY: 0.5176787, DIVERSITY_MEAN_INTER_LIST: 0.8859020, DIVERSITY_HERFINDAHL: 0.9885889, COVERAGE_ITEM: 0.9945698, COVERAGE_ITEM_CORRECT: 0.0581406, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4144223, DIVERSITY_GINI: 0.1478597, SHANNON_ENTROPY: 8.7264715, \n",
      "\n",
      "MF_BPR: Epoch 495 of 1500. Elapsed time 1.64 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.83 seconds. BPR loss 9.70E+00. Sample per second: 14416\n",
      "MF_BPR: Epoch 496 of 1500. Elapsed time 1.65 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.87 seconds. BPR loss 9.78E+00. Sample per second: 14285\n",
      "MF_BPR: Epoch 497 of 1500. Elapsed time 1.65 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.91 seconds. BPR loss 9.71E+00. Sample per second: 14182\n",
      "MF_BPR: Epoch 498 of 1500. Elapsed time 1.65 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.92 seconds. BPR loss 9.75E+00. Sample per second: 14166\n",
      "MF_BPR: Epoch 499 of 1500. Elapsed time 1.65 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.98 seconds. BPR loss 9.73E+00. Sample per second: 13983\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.35 sec. Users per second: 1746\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.59 sec. Users per second: 1759\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2302108, PRECISION: 0.0822546, PRECISION_RECALL_MIN_DEN: 0.1033913, RECALL: 0.0665086, MAP: 0.0473783, MRR: 0.1909510, NDCG: 0.0665776, F1: 0.0735483, HIT_RATE: 0.8225461, ARHR: 0.2679551, NOVELTY: 0.0097627, AVERAGE_POPULARITY: 0.5172540, DIVERSITY_MEAN_INTER_LIST: 0.8863392, DIVERSITY_HERFINDAHL: 0.9886327, COVERAGE_ITEM: 0.9943825, COVERAGE_ITEM_CORRECT: 0.0583279, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4144795, DIVERSITY_GINI: 0.1475674, SHANNON_ENTROPY: 8.7306047, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 500 of 1500. Elapsed time 1.66 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.83 seconds. BPR loss 9.80E+00. Sample per second: 14406\n",
      "MF_BPR: Epoch 501 of 1500. Elapsed time 1.66 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.96 seconds. BPR loss 9.77E+00. Sample per second: 14050\n",
      "MF_BPR: Epoch 502 of 1500. Elapsed time 1.66 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.03 seconds. BPR loss 9.91E+00. Sample per second: 13850\n",
      "MF_BPR: Epoch 503 of 1500. Elapsed time 1.66 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.13 seconds. BPR loss 9.93E+00. Sample per second: 16851\n",
      "MF_BPR: Epoch 504 of 1500. Elapsed time 1.67 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.27 seconds. BPR loss 9.98E+00. Sample per second: 16295\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.28 sec. Users per second: 1684\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.24 sec. Users per second: 1689\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2303476, PRECISION: 0.0822216, PRECISION_RECALL_MIN_DEN: 0.1033667, RECALL: 0.0664895, MAP: 0.0473895, MRR: 0.1909802, NDCG: 0.0665713, F1: 0.0735234, HIT_RATE: 0.8222158, ARHR: 0.2679731, NOVELTY: 0.0097627, AVERAGE_POPULARITY: 0.5167035, DIVERSITY_MEAN_INTER_LIST: 0.8868576, DIVERSITY_HERFINDAHL: 0.9886845, COVERAGE_ITEM: 0.9941953, COVERAGE_ITEM_CORRECT: 0.0582342, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4142506, DIVERSITY_GINI: 0.1472022, SHANNON_ENTROPY: 8.7348153, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 505 of 1500. Elapsed time 1.68 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.67 seconds. BPR loss 9.92E+00. Sample per second: 14903\n",
      "MF_BPR: Epoch 506 of 1500. Elapsed time 1.68 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.78 seconds. BPR loss 9.96E+00. Sample per second: 14565\n",
      "MF_BPR: Epoch 507 of 1500. Elapsed time 1.68 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.21 seconds. BPR loss 1.01E+01. Sample per second: 13362\n",
      "MF_BPR: Epoch 508 of 1500. Elapsed time 1.68 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.28 seconds. BPR loss 1.00E+01. Sample per second: 16262\n",
      "MF_BPR: Epoch 509 of 1500. Elapsed time 1.68 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.38 seconds. BPR loss 1.00E+01. Sample per second: 15885\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.23 sec. Users per second: 1720\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.42 sec. Users per second: 1723\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2302425, PRECISION: 0.0823322, PRECISION_RECALL_MIN_DEN: 0.1034719, RECALL: 0.0665402, MAP: 0.0474350, MRR: 0.1911362, NDCG: 0.0666755, F1: 0.0735986, HIT_RATE: 0.8233216, ARHR: 0.2682232, NOVELTY: 0.0097629, AVERAGE_POPULARITY: 0.5162847, DIVERSITY_MEAN_INTER_LIST: 0.8872059, DIVERSITY_HERFINDAHL: 0.9887193, COVERAGE_ITEM: 0.9943825, COVERAGE_ITEM_CORRECT: 0.0581406, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4147085, DIVERSITY_GINI: 0.1468966, SHANNON_ENTROPY: 8.7386897, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 510 of 1500. Elapsed time 1.69 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.98 seconds. BPR loss 1.01E+01. Sample per second: 13977\n",
      "MF_BPR: Epoch 511 of 1500. Elapsed time 1.70 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.15 seconds. BPR loss 1.01E+01. Sample per second: 13519\n",
      "MF_BPR: Epoch 512 of 1500. Elapsed time 1.70 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.49 seconds. BPR loss 1.01E+01. Sample per second: 15507\n",
      "MF_BPR: Epoch 513 of 1500. Elapsed time 1.70 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.61 seconds. BPR loss 1.02E+01. Sample per second: 15109\n",
      "MF_BPR: Epoch 514 of 1500. Elapsed time 1.70 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.77 seconds. BPR loss 1.02E+01. Sample per second: 14587\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.34 sec. Users per second: 1681\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.22 sec. Users per second: 1689\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2300130, PRECISION: 0.0824169, PRECISION_RECALL_MIN_DEN: 0.1036154, RECALL: 0.0666358, MAP: 0.0474373, MRR: 0.1910766, NDCG: 0.0666810, F1: 0.0736909, HIT_RATE: 0.8241689, ARHR: 0.2682268, NOVELTY: 0.0097626, AVERAGE_POPULARITY: 0.5158849, DIVERSITY_MEAN_INTER_LIST: 0.8875571, DIVERSITY_HERFINDAHL: 0.9887544, COVERAGE_ITEM: 0.9942889, COVERAGE_ITEM_CORRECT: 0.0581406, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4150806, DIVERSITY_GINI: 0.1464914, SHANNON_ENTROPY: 8.7415745, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 515 of 1500. Elapsed time 1.71 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.17 seconds. BPR loss 1.02E+01. Sample per second: 16694\n",
      "MF_BPR: Epoch 516 of 1500. Elapsed time 1.71 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.21 seconds. BPR loss 1.03E+01. Sample per second: 16529\n",
      "MF_BPR: Epoch 517 of 1500. Elapsed time 1.71 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.25 seconds. BPR loss 1.03E+01. Sample per second: 16392\n",
      "MF_BPR: Epoch 518 of 1500. Elapsed time 1.72 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.29 seconds. BPR loss 1.03E+01. Sample per second: 16220\n",
      "MF_BPR: Epoch 519 of 1500. Elapsed time 1.72 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.32 seconds. BPR loss 1.03E+01. Sample per second: 16130\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.50 sec. Users per second: 1672\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.85 sec. Users per second: 1705\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2300733, PRECISION: 0.0824700, PRECISION_RECALL_MIN_DEN: 0.1036399, RECALL: 0.0666092, MAP: 0.0474463, MRR: 0.1909905, NDCG: 0.0666999, F1: 0.0736959, HIT_RATE: 0.8247002, ARHR: 0.2681918, NOVELTY: 0.0097626, AVERAGE_POPULARITY: 0.5154353, DIVERSITY_MEAN_INTER_LIST: 0.8879458, DIVERSITY_HERFINDAHL: 0.9887933, COVERAGE_ITEM: 0.9944762, COVERAGE_ITEM_CORRECT: 0.0583279, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4152094, DIVERSITY_GINI: 0.1462759, SHANNON_ENTROPY: 8.7455091, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 520 of 1500. Elapsed time 1.73 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.36 seconds. BPR loss 1.02E+01. Sample per second: 15971\n",
      "MF_BPR: Epoch 521 of 1500. Elapsed time 1.73 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.41 seconds. BPR loss 1.04E+01. Sample per second: 15775\n",
      "MF_BPR: Epoch 522 of 1500. Elapsed time 1.73 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.48 seconds. BPR loss 1.03E+01. Sample per second: 15549\n",
      "MF_BPR: Epoch 523 of 1500. Elapsed time 1.73 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.55 seconds. BPR loss 1.04E+01. Sample per second: 15295\n",
      "MF_BPR: Epoch 524 of 1500. Elapsed time 1.73 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.62 seconds. BPR loss 1.05E+01. Sample per second: 15059\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.45 sec. Users per second: 1773\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.98 sec. Users per second: 1786\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2301540, PRECISION: 0.0825002, PRECISION_RECALL_MIN_DEN: 0.1037020, RECALL: 0.0666914, MAP: 0.0475114, MRR: 0.1911778, NDCG: 0.0667717, F1: 0.0737582, HIT_RATE: 0.8250018, ARHR: 0.2684487, NOVELTY: 0.0097634, AVERAGE_POPULARITY: 0.5148598, DIVERSITY_MEAN_INTER_LIST: 0.8884439, DIVERSITY_HERFINDAHL: 0.9888431, COVERAGE_ITEM: 0.9940081, COVERAGE_ITEM_CORRECT: 0.0584215, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4152094, DIVERSITY_GINI: 0.1459221, SHANNON_ENTROPY: 8.7499033, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 525 of 1500. Elapsed time 1.75 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.68 seconds. BPR loss 1.04E+01. Sample per second: 14880\n",
      "MF_BPR: Epoch 526 of 1500. Elapsed time 1.75 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.65 seconds. BPR loss 1.05E+01. Sample per second: 14987\n",
      "MF_BPR: Epoch 527 of 1500. Elapsed time 1.75 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.62 seconds. BPR loss 1.04E+01. Sample per second: 15087\n",
      "MF_BPR: Epoch 528 of 1500. Elapsed time 1.75 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.60 seconds. BPR loss 1.05E+01. Sample per second: 15136\n",
      "MF_BPR: Epoch 529 of 1500. Elapsed time 1.75 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.60 seconds. BPR loss 1.05E+01. Sample per second: 15146\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.18 sec. Users per second: 1789\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.80 sec. Users per second: 1795\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2300767, PRECISION: 0.0825404, PRECISION_RECALL_MIN_DEN: 0.1037812, RECALL: 0.0667383, MAP: 0.0475290, MRR: 0.1911139, NDCG: 0.0668129, F1: 0.0738030, HIT_RATE: 0.8254039, ARHR: 0.2685070, NOVELTY: 0.0097632, AVERAGE_POPULARITY: 0.5144576, DIVERSITY_MEAN_INTER_LIST: 0.8888396, DIVERSITY_HERFINDAHL: 0.9888827, COVERAGE_ITEM: 0.9936336, COVERAGE_ITEM_CORRECT: 0.0589832, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4151235, DIVERSITY_GINI: 0.1455892, SHANNON_ENTROPY: 8.7534060, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 530 of 1500. Elapsed time 1.76 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.50 seconds. BPR loss 1.05E+01. Sample per second: 15462\n",
      "MF_BPR: Epoch 531 of 1500. Elapsed time 1.76 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.47 seconds. BPR loss 1.06E+01. Sample per second: 15564\n",
      "MF_BPR: Epoch 532 of 1500. Elapsed time 1.76 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.47 seconds. BPR loss 1.05E+01. Sample per second: 15563\n",
      "MF_BPR: Epoch 533 of 1500. Elapsed time 1.77 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.58 seconds. BPR loss 1.06E+01. Sample per second: 15194\n",
      "MF_BPR: Epoch 534 of 1500. Elapsed time 1.77 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.66 seconds. BPR loss 1.06E+01. Sample per second: 14940\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.38 sec. Users per second: 1745\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.60 sec. Users per second: 1758\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2300616, PRECISION: 0.0826251, PRECISION_RECALL_MIN_DEN: 0.1038680, RECALL: 0.0667703, MAP: 0.0475894, MRR: 0.1913295, NDCG: 0.0669308, F1: 0.0738564, HIT_RATE: 0.8262512, ARHR: 0.2688217, NOVELTY: 0.0097634, AVERAGE_POPULARITY: 0.5137688, DIVERSITY_MEAN_INTER_LIST: 0.8895397, DIVERSITY_HERFINDAHL: 0.9889527, COVERAGE_ITEM: 0.9938208, COVERAGE_ITEM_CORRECT: 0.0587024, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4150806, DIVERSITY_GINI: 0.1453316, SHANNON_ENTROPY: 8.7589884, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 535 of 1500. Elapsed time 1.78 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.41 seconds. BPR loss 1.06E+01. Sample per second: 15772\n",
      "MF_BPR: Epoch 536 of 1500. Elapsed time 1.78 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.52 seconds. BPR loss 1.06E+01. Sample per second: 15413\n",
      "MF_BPR: Epoch 537 of 1500. Elapsed time 1.78 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.67 seconds. BPR loss 1.07E+01. Sample per second: 14908\n",
      "MF_BPR: Epoch 538 of 1500. Elapsed time 1.78 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.64 seconds. BPR loss 1.07E+01. Sample per second: 14996\n",
      "MF_BPR: Epoch 539 of 1500. Elapsed time 1.78 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.68 seconds. BPR loss 1.07E+01. Sample per second: 14871\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.22 sec. Users per second: 1754\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.13 sec. Users per second: 1780\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2297914, PRECISION: 0.0827127, PRECISION_RECALL_MIN_DEN: 0.1039939, RECALL: 0.0668701, MAP: 0.0476067, MRR: 0.1913758, NDCG: 0.0670211, F1: 0.0739525, HIT_RATE: 0.8271272, ARHR: 0.2688579, NOVELTY: 0.0097635, AVERAGE_POPULARITY: 0.5132931, DIVERSITY_MEAN_INTER_LIST: 0.8899478, DIVERSITY_HERFINDAHL: 0.9889935, COVERAGE_ITEM: 0.9940081, COVERAGE_ITEM_CORRECT: 0.0586087, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4151664, DIVERSITY_GINI: 0.1450237, SHANNON_ENTROPY: 8.7622606, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 540 of 1500. Elapsed time 1.80 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.24 seconds. BPR loss 1.08E+01. Sample per second: 13279\n",
      "MF_BPR: Epoch 541 of 1500. Elapsed time 1.80 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.30 seconds. BPR loss 1.08E+01. Sample per second: 16181\n",
      "MF_BPR: Epoch 542 of 1500. Elapsed time 1.80 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 1.08E+01. Sample per second: 15942\n",
      "MF_BPR: Epoch 543 of 1500. Elapsed time 1.80 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.33 seconds. BPR loss 1.09E+01. Sample per second: 16096\n",
      "MF_BPR: Epoch 544 of 1500. Elapsed time 1.80 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 1.08E+01. Sample per second: 15946\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.07 sec. Users per second: 1696\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.03 sec. Users per second: 1657\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2298737, PRECISION: 0.0827759, PRECISION_RECALL_MIN_DEN: 0.1040425, RECALL: 0.0668870, MAP: 0.0476192, MRR: 0.1912612, NDCG: 0.0670268, F1: 0.0739880, HIT_RATE: 0.8277590, ARHR: 0.2688145, NOVELTY: 0.0097635, AVERAGE_POPULARITY: 0.5127303, DIVERSITY_MEAN_INTER_LIST: 0.8904899, DIVERSITY_HERFINDAHL: 0.9890477, COVERAGE_ITEM: 0.9940081, COVERAGE_ITEM_CORRECT: 0.0586087, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4154097, DIVERSITY_GINI: 0.1447792, SHANNON_ENTROPY: 8.7665994, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 545 of 1500. Elapsed time 1.81 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.33 seconds. BPR loss 1.10E+01. Sample per second: 16077\n",
      "MF_BPR: Epoch 546 of 1500. Elapsed time 1.81 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.18 seconds. BPR loss 1.08E+01. Sample per second: 16663\n",
      "MF_BPR: Epoch 547 of 1500. Elapsed time 1.81 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.33 seconds. BPR loss 1.09E+01. Sample per second: 16066\n",
      "MF_BPR: Epoch 548 of 1500. Elapsed time 1.82 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.22 seconds. BPR loss 1.09E+01. Sample per second: 16504\n",
      "MF_BPR: Epoch 549 of 1500. Elapsed time 1.82 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.16 seconds. BPR loss 1.09E+01. Sample per second: 16733\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 47000 ( 67.49% ) in 30.19 sec. Users per second: 1557\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.49 sec. Users per second: 1601\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2295932, PRECISION: 0.0828664, PRECISION_RECALL_MIN_DEN: 0.1041641, RECALL: 0.0669791, MAP: 0.0477166, MRR: 0.1913895, NDCG: 0.0671136, F1: 0.0740805, HIT_RATE: 0.8286637, ARHR: 0.2691165, NOVELTY: 0.0097637, AVERAGE_POPULARITY: 0.5120662, DIVERSITY_MEAN_INTER_LIST: 0.8912020, DIVERSITY_HERFINDAHL: 0.9891189, COVERAGE_ITEM: 0.9940081, COVERAGE_ITEM_CORRECT: 0.0586087, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4154813, DIVERSITY_GINI: 0.1445108, SHANNON_ENTROPY: 8.7711391, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 550 of 1500. Elapsed time 1.83 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.10 seconds. BPR loss 1.10E+01. Sample per second: 13663\n",
      "MF_BPR: Epoch 551 of 1500. Elapsed time 1.83 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 1.10E+01. Sample per second: 15930\n",
      "MF_BPR: Epoch 552 of 1500. Elapsed time 1.83 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.61 seconds. BPR loss 1.10E+01. Sample per second: 15095\n",
      "MF_BPR: Epoch 553 of 1500. Elapsed time 1.83 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.79 seconds. BPR loss 1.10E+01. Sample per second: 14538\n",
      "MF_BPR: Epoch 554 of 1500. Elapsed time 1.83 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.92 seconds. BPR loss 1.10E+01. Sample per second: 14141\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.19 sec. Users per second: 1722\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.23 sec. Users per second: 1731\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2297207, PRECISION: 0.0828535, PRECISION_RECALL_MIN_DEN: 0.1041756, RECALL: 0.0670075, MAP: 0.0476929, MRR: 0.1912295, NDCG: 0.0670577, F1: 0.0740927, HIT_RATE: 0.8285345, ARHR: 0.2689270, NOVELTY: 0.0097638, AVERAGE_POPULARITY: 0.5115662, DIVERSITY_MEAN_INTER_LIST: 0.8916221, DIVERSITY_HERFINDAHL: 0.9891609, COVERAGE_ITEM: 0.9937272, COVERAGE_ITEM_CORRECT: 0.0592641, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4155528, DIVERSITY_GINI: 0.1441190, SHANNON_ENTROPY: 8.7743383, \n",
      "\n",
      "MF_BPR: Epoch 555 of 1500. Elapsed time 1.85 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.39 seconds. BPR loss 1.11E+01. Sample per second: 15854\n",
      "MF_BPR: Epoch 556 of 1500. Elapsed time 1.85 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.52 seconds. BPR loss 1.12E+01. Sample per second: 15395\n",
      "MF_BPR: Epoch 557 of 1500. Elapsed time 1.85 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.65 seconds. BPR loss 1.12E+01. Sample per second: 14980\n",
      "MF_BPR: Epoch 558 of 1500. Elapsed time 1.85 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.79 seconds. BPR loss 1.11E+01. Sample per second: 14548\n",
      "MF_BPR: Epoch 559 of 1500. Elapsed time 1.85 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.03 seconds. BPR loss 1.12E+01. Sample per second: 13851\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.40 sec. Users per second: 1710\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.81 sec. Users per second: 1706\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2295659, PRECISION: 0.0829296, PRECISION_RECALL_MIN_DEN: 0.1043028, RECALL: 0.0671261, MAP: 0.0477063, MRR: 0.1912159, NDCG: 0.0670662, F1: 0.0741957, HIT_RATE: 0.8292956, ARHR: 0.2689930, NOVELTY: 0.0097638, AVERAGE_POPULARITY: 0.5110372, DIVERSITY_MEAN_INTER_LIST: 0.8921543, DIVERSITY_HERFINDAHL: 0.9892142, COVERAGE_ITEM: 0.9937272, COVERAGE_ITEM_CORRECT: 0.0588896, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4157389, DIVERSITY_GINI: 0.1438683, SHANNON_ENTROPY: 8.7784698, \n",
      "\n",
      "MF_BPR: Epoch 560 of 1500. Elapsed time 1.86 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.16 seconds. BPR loss 1.12E+01. Sample per second: 13500\n",
      "MF_BPR: Epoch 561 of 1500. Elapsed time 1.87 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.39 seconds. BPR loss 1.13E+01. Sample per second: 15854\n",
      "MF_BPR: Epoch 562 of 1500. Elapsed time 1.87 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.55 seconds. BPR loss 1.13E+01. Sample per second: 15291\n",
      "MF_BPR: Epoch 563 of 1500. Elapsed time 1.87 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.75 seconds. BPR loss 1.13E+01. Sample per second: 14661\n",
      "MF_BPR: Epoch 564 of 1500. Elapsed time 1.87 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.95 seconds. BPR loss 1.13E+01. Sample per second: 14081\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.37 sec. Users per second: 1646\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.98 sec. Users per second: 1659\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2298506, PRECISION: 0.0828333, PRECISION_RECALL_MIN_DEN: 0.1042216, RECALL: 0.0670969, MAP: 0.0476771, MRR: 0.1912453, NDCG: 0.0670439, F1: 0.0741393, HIT_RATE: 0.8283335, ARHR: 0.2689364, NOVELTY: 0.0097645, AVERAGE_POPULARITY: 0.5103189, DIVERSITY_MEAN_INTER_LIST: 0.8926914, DIVERSITY_HERFINDAHL: 0.9892679, COVERAGE_ITEM: 0.9932591, COVERAGE_ITEM_CORRECT: 0.0590769, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4157389, DIVERSITY_GINI: 0.1435559, SHANNON_ENTROPY: 8.7830969, \n",
      "\n",
      "MF_BPR: Epoch 565 of 1500. Elapsed time 1.88 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.17 seconds. BPR loss 1.13E+01. Sample per second: 13479\n",
      "MF_BPR: Epoch 566 of 1500. Elapsed time 1.88 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.33 seconds. BPR loss 1.13E+01. Sample per second: 16095\n",
      "MF_BPR: Epoch 567 of 1500. Elapsed time 1.88 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.50 seconds. BPR loss 1.14E+01. Sample per second: 15464\n",
      "MF_BPR: Epoch 568 of 1500. Elapsed time 1.89 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.68 seconds. BPR loss 1.13E+01. Sample per second: 14863\n",
      "MF_BPR: Epoch 569 of 1500. Elapsed time 1.89 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.84 seconds. BPR loss 1.14E+01. Sample per second: 14388\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.43 sec. Users per second: 1643\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.95 sec. Users per second: 1660\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2297060, PRECISION: 0.0828994, PRECISION_RECALL_MIN_DEN: 0.1043043, RECALL: 0.0671559, MAP: 0.0477572, MRR: 0.1913830, NDCG: 0.0671390, F1: 0.0742018, HIT_RATE: 0.8289940, ARHR: 0.2691682, NOVELTY: 0.0097650, AVERAGE_POPULARITY: 0.5096896, DIVERSITY_MEAN_INTER_LIST: 0.8933231, DIVERSITY_HERFINDAHL: 0.9893310, COVERAGE_ITEM: 0.9932591, COVERAGE_ITEM_CORRECT: 0.0587024, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4156959, DIVERSITY_GINI: 0.1431997, SHANNON_ENTROPY: 8.7880840, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 570 of 1500. Elapsed time 1.90 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.06 seconds. BPR loss 1.14E+01. Sample per second: 13774\n",
      "MF_BPR: Epoch 571 of 1500. Elapsed time 1.90 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.15 seconds. BPR loss 1.14E+01. Sample per second: 16764\n",
      "MF_BPR: Epoch 572 of 1500. Elapsed time 1.90 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.22 seconds. BPR loss 1.15E+01. Sample per second: 16500\n",
      "MF_BPR: Epoch 573 of 1500. Elapsed time 1.90 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.31 seconds. BPR loss 1.14E+01. Sample per second: 16168\n",
      "MF_BPR: Epoch 574 of 1500. Elapsed time 1.90 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.36 seconds. BPR loss 1.15E+01. Sample per second: 15969\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.14 sec. Users per second: 1659\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.37 sec. Users per second: 1683\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2298304, PRECISION: 0.0829267, PRECISION_RECALL_MIN_DEN: 0.1043273, RECALL: 0.0671443, MAP: 0.0478090, MRR: 0.1914032, NDCG: 0.0671380, F1: 0.0742056, HIT_RATE: 0.8292669, ARHR: 0.2693036, NOVELTY: 0.0097651, AVERAGE_POPULARITY: 0.5092264, DIVERSITY_MEAN_INTER_LIST: 0.8937752, DIVERSITY_HERFINDAHL: 0.9893762, COVERAGE_ITEM: 0.9930718, COVERAGE_ITEM_CORRECT: 0.0587960, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4155385, DIVERSITY_GINI: 0.1428433, SHANNON_ENTROPY: 8.7914509, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 575 of 1500. Elapsed time 1.92 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.97 seconds. BPR loss 1.16E+01. Sample per second: 14016\n",
      "MF_BPR: Epoch 576 of 1500. Elapsed time 1.92 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.12 seconds. BPR loss 1.15E+01. Sample per second: 13613\n",
      "MF_BPR: Epoch 577 of 1500. Elapsed time 1.92 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.21 seconds. BPR loss 1.16E+01. Sample per second: 16531\n",
      "MF_BPR: Epoch 578 of 1500. Elapsed time 1.92 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.32 seconds. BPR loss 1.16E+01. Sample per second: 16130\n",
      "MF_BPR: Epoch 579 of 1500. Elapsed time 1.92 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 1.16E+01. Sample per second: 15928\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.14 sec. Users per second: 1692\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.77 sec. Users per second: 1708\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2297249, PRECISION: 0.0829554, PRECISION_RECALL_MIN_DEN: 0.1043512, RECALL: 0.0671407, MAP: 0.0478071, MRR: 0.1913575, NDCG: 0.0671156, F1: 0.0742149, HIT_RATE: 0.8295541, ARHR: 0.2692163, NOVELTY: 0.0097651, AVERAGE_POPULARITY: 0.5087930, DIVERSITY_MEAN_INTER_LIST: 0.8940347, DIVERSITY_HERFINDAHL: 0.9894022, COVERAGE_ITEM: 0.9928846, COVERAGE_ITEM_CORRECT: 0.0587960, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4156530, DIVERSITY_GINI: 0.1425439, SHANNON_ENTROPY: 8.7933571, \n",
      "\n",
      "MF_BPR: Epoch 580 of 1500. Elapsed time 1.93 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.36 seconds. BPR loss 1.16E+01. Sample per second: 15985\n",
      "MF_BPR: Epoch 581 of 1500. Elapsed time 1.93 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.62 seconds. BPR loss 1.17E+01. Sample per second: 15068\n",
      "MF_BPR: Epoch 582 of 1500. Elapsed time 1.94 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.79 seconds. BPR loss 1.17E+01. Sample per second: 14528\n",
      "MF_BPR: Epoch 583 of 1500. Elapsed time 1.94 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.93 seconds. BPR loss 1.18E+01. Sample per second: 14122\n",
      "MF_BPR: Epoch 584 of 1500. Elapsed time 1.94 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.30 seconds. BPR loss 1.18E+01. Sample per second: 13132\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.93% ) in 30.02 sec. Users per second: 1599\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.82 sec. Users per second: 1626\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2296625, PRECISION: 0.0829827, PRECISION_RECALL_MIN_DEN: 0.1043972, RECALL: 0.0671918, MAP: 0.0478263, MRR: 0.1913237, NDCG: 0.0671294, F1: 0.0742570, HIT_RATE: 0.8298270, ARHR: 0.2692830, NOVELTY: 0.0097655, AVERAGE_POPULARITY: 0.5082652, DIVERSITY_MEAN_INTER_LIST: 0.8946029, DIVERSITY_HERFINDAHL: 0.9894590, COVERAGE_ITEM: 0.9927909, COVERAGE_ITEM_CORRECT: 0.0586087, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4155814, DIVERSITY_GINI: 0.1421852, SHANNON_ENTROPY: 8.7971754, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 585 of 1500. Elapsed time 1.95 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.34 seconds. BPR loss 1.17E+01. Sample per second: 16041\n",
      "MF_BPR: Epoch 586 of 1500. Elapsed time 1.95 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.39 seconds. BPR loss 1.17E+01. Sample per second: 15870\n",
      "MF_BPR: Epoch 587 of 1500. Elapsed time 1.95 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.60 seconds. BPR loss 1.18E+01. Sample per second: 15125\n",
      "MF_BPR: Epoch 588 of 1500. Elapsed time 1.95 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.86 seconds. BPR loss 1.17E+01. Sample per second: 14329\n",
      "MF_BPR: Epoch 589 of 1500. Elapsed time 1.96 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.02 seconds. BPR loss 1.18E+01. Sample per second: 13859\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.23 sec. Users per second: 1654\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.76 sec. Users per second: 1668\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2296604, PRECISION: 0.0830689, PRECISION_RECALL_MIN_DEN: 0.1045010, RECALL: 0.0672580, MAP: 0.0478782, MRR: 0.1913464, NDCG: 0.0671493, F1: 0.0743319, HIT_RATE: 0.8306886, ARHR: 0.2694208, NOVELTY: 0.0097660, AVERAGE_POPULARITY: 0.5075245, DIVERSITY_MEAN_INTER_LIST: 0.8952606, DIVERSITY_HERFINDAHL: 0.9895248, COVERAGE_ITEM: 0.9930718, COVERAGE_ITEM_CORRECT: 0.0589832, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4157818, DIVERSITY_GINI: 0.1419404, SHANNON_ENTROPY: 8.8024985, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 590 of 1500. Elapsed time 1.97 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.05 seconds. BPR loss 1.18E+01. Sample per second: 13802\n",
      "MF_BPR: Epoch 591 of 1500. Elapsed time 1.97 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.19 seconds. BPR loss 1.18E+01. Sample per second: 16620\n",
      "MF_BPR: Epoch 592 of 1500. Elapsed time 1.97 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.32 seconds. BPR loss 1.18E+01. Sample per second: 16129\n",
      "MF_BPR: Epoch 593 of 1500. Elapsed time 1.97 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.46 seconds. BPR loss 1.19E+01. Sample per second: 15622\n",
      "MF_BPR: Epoch 594 of 1500. Elapsed time 1.97 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.59 seconds. BPR loss 1.19E+01. Sample per second: 15173\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.45 sec. Users per second: 1675\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.33 sec. Users per second: 1685\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2298841, PRECISION: 0.0830272, PRECISION_RECALL_MIN_DEN: 0.1044334, RECALL: 0.0672007, MAP: 0.0478904, MRR: 0.1913343, NDCG: 0.0671621, F1: 0.0742803, HIT_RATE: 0.8302721, ARHR: 0.2694112, NOVELTY: 0.0097668, AVERAGE_POPULARITY: 0.5069354, DIVERSITY_MEAN_INTER_LIST: 0.8955995, DIVERSITY_HERFINDAHL: 0.9895587, COVERAGE_ITEM: 0.9933527, COVERAGE_ITEM_CORRECT: 0.0592641, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4155099, DIVERSITY_GINI: 0.1415554, SHANNON_ENTROPY: 8.8049506, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 595 of 1500. Elapsed time 1.99 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.22 seconds. BPR loss 1.19E+01. Sample per second: 16497\n",
      "MF_BPR: Epoch 596 of 1500. Elapsed time 1.99 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.39 seconds. BPR loss 1.19E+01. Sample per second: 15850\n",
      "MF_BPR: Epoch 597 of 1500. Elapsed time 1.99 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.56 seconds. BPR loss 1.20E+01. Sample per second: 15266\n",
      "MF_BPR: Epoch 598 of 1500. Elapsed time 1.99 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.73 seconds. BPR loss 1.21E+01. Sample per second: 14733\n",
      "MF_BPR: Epoch 599 of 1500. Elapsed time 1.99 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.94 seconds. BPR loss 1.20E+01. Sample per second: 14103\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.39 sec. Users per second: 1613\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.15 sec. Users per second: 1614\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2296843, PRECISION: 0.0830660, PRECISION_RECALL_MIN_DEN: 0.1045219, RECALL: 0.0672672, MAP: 0.0479003, MRR: 0.1912826, NDCG: 0.0671851, F1: 0.0743364, HIT_RATE: 0.8306599, ARHR: 0.2693923, NOVELTY: 0.0097672, AVERAGE_POPULARITY: 0.5062233, DIVERSITY_MEAN_INTER_LIST: 0.8963092, DIVERSITY_HERFINDAHL: 0.9896296, COVERAGE_ITEM: 0.9929782, COVERAGE_ITEM_CORRECT: 0.0596386, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4155242, DIVERSITY_GINI: 0.1412750, SHANNON_ENTROPY: 8.8101496, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 600 of 1500. Elapsed time 2.00 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.42 seconds. BPR loss 1.19E+01. Sample per second: 15762\n",
      "MF_BPR: Epoch 601 of 1500. Elapsed time 2.00 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.61 seconds. BPR loss 1.20E+01. Sample per second: 15115\n",
      "MF_BPR: Epoch 602 of 1500. Elapsed time 2.01 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.78 seconds. BPR loss 1.21E+01. Sample per second: 14567\n",
      "MF_BPR: Epoch 603 of 1500. Elapsed time 2.01 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.03 seconds. BPR loss 1.20E+01. Sample per second: 13839\n",
      "MF_BPR: Epoch 604 of 1500. Elapsed time 2.01 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.21 seconds. BPR loss 1.22E+01. Sample per second: 16549\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.05 sec. Users per second: 1664\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.46 sec. Users per second: 1679\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2296741, PRECISION: 0.0830904, PRECISION_RECALL_MIN_DEN: 0.1045327, RECALL: 0.0672711, MAP: 0.0479343, MRR: 0.1912967, NDCG: 0.0671591, F1: 0.0743486, HIT_RATE: 0.8309040, ARHR: 0.2694542, NOVELTY: 0.0097680, AVERAGE_POPULARITY: 0.5054966, DIVERSITY_MEAN_INTER_LIST: 0.8969640, DIVERSITY_HERFINDAHL: 0.9896951, COVERAGE_ITEM: 0.9930718, COVERAGE_ITEM_CORRECT: 0.0595450, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4152523, DIVERSITY_GINI: 0.1409980, SHANNON_ENTROPY: 8.8148767, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 605 of 1500. Elapsed time 2.02 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.92 seconds. BPR loss 1.21E+01. Sample per second: 14155\n",
      "MF_BPR: Epoch 606 of 1500. Elapsed time 2.02 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.02 seconds. BPR loss 1.21E+01. Sample per second: 13871\n",
      "MF_BPR: Epoch 607 of 1500. Elapsed time 2.02 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.13 seconds. BPR loss 1.21E+01. Sample per second: 16872\n",
      "MF_BPR: Epoch 608 of 1500. Elapsed time 2.02 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.26 seconds. BPR loss 1.22E+01. Sample per second: 16358\n",
      "MF_BPR: Epoch 609 of 1500. Elapsed time 2.03 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.37 seconds. BPR loss 1.22E+01. Sample per second: 15932\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.46 sec. Users per second: 1707\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.66 sec. Users per second: 1713\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2296916, PRECISION: 0.0830818, PRECISION_RECALL_MIN_DEN: 0.1045299, RECALL: 0.0672614, MAP: 0.0479389, MRR: 0.1911890, NDCG: 0.0671275, F1: 0.0743392, HIT_RATE: 0.8308178, ARHR: 0.2694009, NOVELTY: 0.0097684, AVERAGE_POPULARITY: 0.5049314, DIVERSITY_MEAN_INTER_LIST: 0.8975042, DIVERSITY_HERFINDAHL: 0.9897491, COVERAGE_ITEM: 0.9928846, COVERAGE_ITEM_CORRECT: 0.0597322, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4151807, DIVERSITY_GINI: 0.1407327, SHANNON_ENTROPY: 8.8191556, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 610 of 1500. Elapsed time 2.04 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.29 seconds. BPR loss 1.22E+01. Sample per second: 16245\n",
      "MF_BPR: Epoch 611 of 1500. Elapsed time 2.04 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.48 seconds. BPR loss 1.22E+01. Sample per second: 15556\n",
      "MF_BPR: Epoch 612 of 1500. Elapsed time 2.04 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.68 seconds. BPR loss 1.22E+01. Sample per second: 14888\n",
      "MF_BPR: Epoch 613 of 1500. Elapsed time 2.04 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.85 seconds. BPR loss 1.23E+01. Sample per second: 14350\n",
      "MF_BPR: Epoch 614 of 1500. Elapsed time 2.04 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.07 seconds. BPR loss 1.24E+01. Sample per second: 13731\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.93% ) in 30.25 sec. Users per second: 1587\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.43 sec. Users per second: 1603\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2296823, PRECISION: 0.0831407, PRECISION_RECALL_MIN_DEN: 0.1046169, RECALL: 0.0673336, MAP: 0.0479612, MRR: 0.1912250, NDCG: 0.0671456, F1: 0.0744069, HIT_RATE: 0.8314066, ARHR: 0.2694841, NOVELTY: 0.0097686, AVERAGE_POPULARITY: 0.5043343, DIVERSITY_MEAN_INTER_LIST: 0.8980773, DIVERSITY_HERFINDAHL: 0.9898064, COVERAGE_ITEM: 0.9930718, COVERAGE_ITEM_CORRECT: 0.0595450, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4154526, DIVERSITY_GINI: 0.1403732, SHANNON_ENTROPY: 8.8233329, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 615 of 1500. Elapsed time 2.06 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.82 seconds. BPR loss 1.23E+01. Sample per second: 14451\n",
      "MF_BPR: Epoch 616 of 1500. Elapsed time 2.06 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.08 seconds. BPR loss 1.24E+01. Sample per second: 13719\n",
      "MF_BPR: Epoch 617 of 1500. Elapsed time 2.06 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.34 seconds. BPR loss 1.24E+01. Sample per second: 16027\n",
      "MF_BPR: Epoch 618 of 1500. Elapsed time 2.06 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.60 seconds. BPR loss 1.23E+01. Sample per second: 15143\n",
      "MF_BPR: Epoch 619 of 1500. Elapsed time 2.06 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.81 seconds. BPR loss 1.24E+01. Sample per second: 14465\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.44 sec. Users per second: 1610\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.57 sec. Users per second: 1636\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2296771, PRECISION: 0.0831895, PRECISION_RECALL_MIN_DEN: 0.1046720, RECALL: 0.0673883, MAP: 0.0480020, MRR: 0.1913861, NDCG: 0.0671682, F1: 0.0744598, HIT_RATE: 0.8318949, ARHR: 0.2696964, NOVELTY: 0.0097688, AVERAGE_POPULARITY: 0.5038087, DIVERSITY_MEAN_INTER_LIST: 0.8985875, DIVERSITY_HERFINDAHL: 0.9898575, COVERAGE_ITEM: 0.9930718, COVERAGE_ITEM_CORRECT: 0.0598259, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4154526, DIVERSITY_GINI: 0.1400709, SHANNON_ENTROPY: 8.8274369, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 620 of 1500. Elapsed time 2.07 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.68 seconds. BPR loss 1.24E+01. Sample per second: 14878\n",
      "MF_BPR: Epoch 621 of 1500. Elapsed time 2.08 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.86 seconds. BPR loss 1.24E+01. Sample per second: 14317\n",
      "MF_BPR: Epoch 622 of 1500. Elapsed time 2.08 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.06 seconds. BPR loss 1.24E+01. Sample per second: 13763\n",
      "MF_BPR: Epoch 623 of 1500. Elapsed time 2.08 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.24 seconds. BPR loss 1.24E+01. Sample per second: 16415\n",
      "MF_BPR: Epoch 624 of 1500. Elapsed time 2.08 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.43 seconds. BPR loss 1.25E+01. Sample per second: 15715\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.56 sec. Users per second: 1636\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.80 sec. Users per second: 1627\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2297213, PRECISION: 0.0832297, PRECISION_RECALL_MIN_DEN: 0.1047083, RECALL: 0.0674128, MAP: 0.0480349, MRR: 0.1914069, NDCG: 0.0672145, F1: 0.0744909, HIT_RATE: 0.8322970, ARHR: 0.2697931, NOVELTY: 0.0097690, AVERAGE_POPULARITY: 0.5032613, DIVERSITY_MEAN_INTER_LIST: 0.8989922, DIVERSITY_HERFINDAHL: 0.9898979, COVERAGE_ITEM: 0.9926973, COVERAGE_ITEM_CORRECT: 0.0595450, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4154240, DIVERSITY_GINI: 0.1397597, SHANNON_ENTROPY: 8.8313319, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 625 of 1500. Elapsed time 2.09 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.57 seconds. BPR loss 1.25E+01. Sample per second: 15223\n",
      "MF_BPR: Epoch 626 of 1500. Elapsed time 2.09 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.79 seconds. BPR loss 1.25E+01. Sample per second: 14552\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 627 of 1500. Elapsed time 2.09 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.01 seconds. BPR loss 1.25E+01. Sample per second: 13899\n",
      "MF_BPR: Epoch 628 of 1500. Elapsed time 2.10 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.23 seconds. BPR loss 1.25E+01. Sample per second: 16470\n",
      "MF_BPR: Epoch 629 of 1500. Elapsed time 2.10 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.40 seconds. BPR loss 1.26E+01. Sample per second: 15832\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.43 sec. Users per second: 1709\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.59 sec. Users per second: 1716\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2295355, PRECISION: 0.0832871, PRECISION_RECALL_MIN_DEN: 0.1047324, RECALL: 0.0674161, MAP: 0.0480533, MRR: 0.1912169, NDCG: 0.0671883, F1: 0.0745159, HIT_RATE: 0.8328714, ARHR: 0.2697041, NOVELTY: 0.0097695, AVERAGE_POPULARITY: 0.5027025, DIVERSITY_MEAN_INTER_LIST: 0.8994454, DIVERSITY_HERFINDAHL: 0.9899432, COVERAGE_ITEM: 0.9927909, COVERAGE_ITEM_CORRECT: 0.0594514, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4152094, DIVERSITY_GINI: 0.1395183, SHANNON_ENTROPY: 8.8347775, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 630 of 1500. Elapsed time 2.11 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.26 seconds. BPR loss 1.27E+01. Sample per second: 16360\n",
      "MF_BPR: Epoch 631 of 1500. Elapsed time 2.11 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.35 seconds. BPR loss 1.25E+01. Sample per second: 16023\n",
      "MF_BPR: Epoch 632 of 1500. Elapsed time 2.11 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.47 seconds. BPR loss 1.26E+01. Sample per second: 15563\n",
      "MF_BPR: Epoch 633 of 1500. Elapsed time 2.11 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.65 seconds. BPR loss 1.27E+01. Sample per second: 14980\n",
      "MF_BPR: Epoch 634 of 1500. Elapsed time 2.11 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.87 seconds. BPR loss 1.27E+01. Sample per second: 14285\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.49 sec. Users per second: 1640\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.20 sec. Users per second: 1650\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2293173, PRECISION: 0.0833489, PRECISION_RECALL_MIN_DEN: 0.1048400, RECALL: 0.0674931, MAP: 0.0480458, MRR: 0.1910168, NDCG: 0.0672329, F1: 0.0745876, HIT_RATE: 0.8334889, ARHR: 0.2694934, NOVELTY: 0.0097699, AVERAGE_POPULARITY: 0.5020688, DIVERSITY_MEAN_INTER_LIST: 0.8999017, DIVERSITY_HERFINDAHL: 0.9899889, COVERAGE_ITEM: 0.9926973, COVERAGE_ITEM_CORRECT: 0.0593577, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4155814, DIVERSITY_GINI: 0.1391993, SHANNON_ENTROPY: 8.8381576, \n",
      "\n",
      "MF_BPR: Epoch 635 of 1500. Elapsed time 2.13 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.33 seconds. BPR loss 1.26E+01. Sample per second: 16080\n",
      "MF_BPR: Epoch 636 of 1500. Elapsed time 2.13 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.51 seconds. BPR loss 1.27E+01. Sample per second: 15430\n",
      "MF_BPR: Epoch 637 of 1500. Elapsed time 2.13 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.73 seconds. BPR loss 1.27E+01. Sample per second: 14716\n",
      "MF_BPR: Epoch 638 of 1500. Elapsed time 2.13 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.93 seconds. BPR loss 1.27E+01. Sample per second: 14113\n",
      "MF_BPR: Epoch 639 of 1500. Elapsed time 2.13 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.30 seconds. BPR loss 1.26E+01. Sample per second: 13150\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.53 sec. Users per second: 1638\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.80 sec. Users per second: 1627\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2291776, PRECISION: 0.0833072, PRECISION_RECALL_MIN_DEN: 0.1047704, RECALL: 0.0674561, MAP: 0.0480292, MRR: 0.1909007, NDCG: 0.0672091, F1: 0.0745484, HIT_RATE: 0.8330724, ARHR: 0.2693880, NOVELTY: 0.0097700, AVERAGE_POPULARITY: 0.5015754, DIVERSITY_MEAN_INTER_LIST: 0.9003490, DIVERSITY_HERFINDAHL: 0.9900336, COVERAGE_ITEM: 0.9928846, COVERAGE_ITEM_CORRECT: 0.0595450, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4153095, DIVERSITY_GINI: 0.1389361, SHANNON_ENTROPY: 8.8417918, \n",
      "\n",
      "MF_BPR: Epoch 640 of 1500. Elapsed time 2.14 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.56 seconds. BPR loss 1.28E+01. Sample per second: 15257\n",
      "MF_BPR: Epoch 641 of 1500. Elapsed time 2.15 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.78 seconds. BPR loss 1.28E+01. Sample per second: 14577\n",
      "MF_BPR: Epoch 642 of 1500. Elapsed time 2.15 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.97 seconds. BPR loss 1.28E+01. Sample per second: 14003\n",
      "MF_BPR: Epoch 643 of 1500. Elapsed time 2.15 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.16 seconds. BPR loss 1.29E+01. Sample per second: 13483\n",
      "MF_BPR: Epoch 644 of 1500. Elapsed time 2.15 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.29 seconds. BPR loss 1.28E+01. Sample per second: 16226\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.44 sec. Users per second: 1708\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.86 sec. Users per second: 1704\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2291795, PRECISION: 0.0833862, PRECISION_RECALL_MIN_DEN: 0.1049242, RECALL: 0.0675929, MAP: 0.0480451, MRR: 0.1908352, NDCG: 0.0672165, F1: 0.0746635, HIT_RATE: 0.8338623, ARHR: 0.2694080, NOVELTY: 0.0097707, AVERAGE_POPULARITY: 0.5009103, DIVERSITY_MEAN_INTER_LIST: 0.9009346, DIVERSITY_HERFINDAHL: 0.9900922, COVERAGE_ITEM: 0.9926037, COVERAGE_ITEM_CORRECT: 0.0594514, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4157245, DIVERSITY_GINI: 0.1387251, SHANNON_ENTROPY: 8.8473572, \n",
      "\n",
      "MF_BPR: Epoch 645 of 1500. Elapsed time 2.16 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.42 seconds. BPR loss 1.29E+01. Sample per second: 15740\n",
      "MF_BPR: Epoch 646 of 1500. Elapsed time 2.16 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.69 seconds. BPR loss 1.28E+01. Sample per second: 14857\n",
      "MF_BPR: Epoch 647 of 1500. Elapsed time 2.16 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.96 seconds. BPR loss 1.29E+01. Sample per second: 14032\n",
      "MF_BPR: Epoch 648 of 1500. Elapsed time 2.16 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.17 seconds. BPR loss 1.29E+01. Sample per second: 13457\n",
      "MF_BPR: Epoch 649 of 1500. Elapsed time 2.17 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.40 seconds. BPR loss 1.28E+01. Sample per second: 15816\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.93% ) in 30.28 sec. Users per second: 1585\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.81 sec. Users per second: 1627\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2293025, PRECISION: 0.0834666, PRECISION_RECALL_MIN_DEN: 0.1049855, RECALL: 0.0676287, MAP: 0.0481055, MRR: 0.1911708, NDCG: 0.0673038, F1: 0.0747176, HIT_RATE: 0.8346665, ARHR: 0.2698483, NOVELTY: 0.0097705, AVERAGE_POPULARITY: 0.5004211, DIVERSITY_MEAN_INTER_LIST: 0.9013304, DIVERSITY_HERFINDAHL: 0.9901317, COVERAGE_ITEM: 0.9926037, COVERAGE_ITEM_CORRECT: 0.0594514, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4158390, DIVERSITY_GINI: 0.1384665, SHANNON_ENTROPY: 8.8509120, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 650 of 1500. Elapsed time 2.18 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.53 seconds. BPR loss 1.30E+01. Sample per second: 15362\n",
      "MF_BPR: Epoch 651 of 1500. Elapsed time 2.18 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.75 seconds. BPR loss 1.30E+01. Sample per second: 14647\n",
      "MF_BPR: Epoch 652 of 1500. Elapsed time 2.18 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.97 seconds. BPR loss 1.30E+01. Sample per second: 14015\n",
      "MF_BPR: Epoch 653 of 1500. Elapsed time 2.18 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.16 seconds. BPR loss 1.29E+01. Sample per second: 13505\n",
      "MF_BPR: Epoch 654 of 1500. Elapsed time 2.18 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.38 seconds. BPR loss 1.31E+01. Sample per second: 15900\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.10 sec. Users per second: 1661\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.54 sec. Users per second: 1676\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2294309, PRECISION: 0.0834379, PRECISION_RECALL_MIN_DEN: 0.1049231, RECALL: 0.0675701, MAP: 0.0481448, MRR: 0.1911620, NDCG: 0.0672965, F1: 0.0746703, HIT_RATE: 0.8343793, ARHR: 0.2699014, NOVELTY: 0.0097714, AVERAGE_POPULARITY: 0.4995895, DIVERSITY_MEAN_INTER_LIST: 0.9020277, DIVERSITY_HERFINDAHL: 0.9902015, COVERAGE_ITEM: 0.9922292, COVERAGE_ITEM_CORRECT: 0.0592641, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4156530, DIVERSITY_GINI: 0.1381395, SHANNON_ENTROPY: 8.8560404, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 655 of 1500. Elapsed time 2.20 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.19 seconds. BPR loss 1.30E+01. Sample per second: 16632\n",
      "MF_BPR: Epoch 656 of 1500. Elapsed time 2.20 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.32 seconds. BPR loss 1.31E+01. Sample per second: 16127\n",
      "MF_BPR: Epoch 657 of 1500. Elapsed time 2.20 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.47 seconds. BPR loss 1.31E+01. Sample per second: 15566\n",
      "MF_BPR: Epoch 658 of 1500. Elapsed time 2.20 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.64 seconds. BPR loss 1.31E+01. Sample per second: 15014\n",
      "MF_BPR: Epoch 659 of 1500. Elapsed time 2.20 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.85 seconds. BPR loss 1.32E+01. Sample per second: 14363\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.24 sec. Users per second: 1654\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.44 sec. Users per second: 1680\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2295937, PRECISION: 0.0833604, PRECISION_RECALL_MIN_DEN: 0.1049040, RECALL: 0.0676203, MAP: 0.0481198, MRR: 0.1911248, NDCG: 0.0672944, F1: 0.0746699, HIT_RATE: 0.8336038, ARHR: 0.2697472, NOVELTY: 0.0097717, AVERAGE_POPULARITY: 0.4990845, DIVERSITY_MEAN_INTER_LIST: 0.9023564, DIVERSITY_HERFINDAHL: 0.9902343, COVERAGE_ITEM: 0.9922292, COVERAGE_ITEM_CORRECT: 0.0593577, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4157245, DIVERSITY_GINI: 0.1379352, SHANNON_ENTROPY: 8.8592935, \n",
      "\n",
      "MF_BPR: Epoch 660 of 1500. Elapsed time 2.21 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.47 seconds. BPR loss 1.31E+01. Sample per second: 15583\n",
      "MF_BPR: Epoch 661 of 1500. Elapsed time 2.22 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.59 seconds. BPR loss 1.31E+01. Sample per second: 15180\n",
      "MF_BPR: Epoch 662 of 1500. Elapsed time 2.22 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.69 seconds. BPR loss 1.31E+01. Sample per second: 14840\n",
      "MF_BPR: Epoch 663 of 1500. Elapsed time 2.22 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.82 seconds. BPR loss 1.32E+01. Sample per second: 14460\n",
      "MF_BPR: Epoch 664 of 1500. Elapsed time 2.22 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.94 seconds. BPR loss 1.32E+01. Sample per second: 14094\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.13 sec. Users per second: 1693\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.10 sec. Users per second: 1694\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2294370, PRECISION: 0.0833834, PRECISION_RECALL_MIN_DEN: 0.1048917, RECALL: 0.0675954, MAP: 0.0480795, MRR: 0.1908752, NDCG: 0.0672552, F1: 0.0746639, HIT_RATE: 0.8338336, ARHR: 0.2695159, NOVELTY: 0.0097713, AVERAGE_POPULARITY: 0.4987890, DIVERSITY_MEAN_INTER_LIST: 0.9026005, DIVERSITY_HERFINDAHL: 0.9902588, COVERAGE_ITEM: 0.9916674, COVERAGE_ITEM_CORRECT: 0.0593577, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4155814, DIVERSITY_GINI: 0.1375280, SHANNON_ENTROPY: 8.8604991, \n",
      "\n",
      "MF_BPR: Epoch 665 of 1500. Elapsed time 2.23 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.32 seconds. BPR loss 1.32E+01. Sample per second: 16125\n",
      "MF_BPR: Epoch 666 of 1500. Elapsed time 2.23 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.55 seconds. BPR loss 1.32E+01. Sample per second: 15301\n",
      "MF_BPR: Epoch 667 of 1500. Elapsed time 2.23 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.79 seconds. BPR loss 1.32E+01. Sample per second: 14550\n",
      "MF_BPR: Epoch 668 of 1500. Elapsed time 2.23 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.95 seconds. BPR loss 1.32E+01. Sample per second: 14068\n",
      "MF_BPR: Epoch 669 of 1500. Elapsed time 2.24 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.11 seconds. BPR loss 1.33E+01. Sample per second: 13628\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.93% ) in 30.53 sec. Users per second: 1572\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.21 sec. Users per second: 1612\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2295450, PRECISION: 0.0834207, PRECISION_RECALL_MIN_DEN: 0.1049891, RECALL: 0.0676938, MAP: 0.0481041, MRR: 0.1909093, NDCG: 0.0673268, F1: 0.0747389, HIT_RATE: 0.8342069, ARHR: 0.2696169, NOVELTY: 0.0097709, AVERAGE_POPULARITY: 0.4985670, DIVERSITY_MEAN_INTER_LIST: 0.9027153, DIVERSITY_HERFINDAHL: 0.9902702, COVERAGE_ITEM: 0.9919483, COVERAGE_ITEM_CORRECT: 0.0593577, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4158104, DIVERSITY_GINI: 0.1372980, SHANNON_ENTROPY: 8.8623349, \n",
      "\n",
      "MF_BPR: Epoch 670 of 1500. Elapsed time 2.25 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.53 seconds. BPR loss 1.33E+01. Sample per second: 15359\n",
      "MF_BPR: Epoch 671 of 1500. Elapsed time 2.25 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.68 seconds. BPR loss 1.33E+01. Sample per second: 14882\n",
      "MF_BPR: Epoch 672 of 1500. Elapsed time 2.25 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.78 seconds. BPR loss 1.33E+01. Sample per second: 14577\n",
      "MF_BPR: Epoch 673 of 1500. Elapsed time 2.25 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.88 seconds. BPR loss 1.33E+01. Sample per second: 14258\n",
      "MF_BPR: Epoch 674 of 1500. Elapsed time 2.25 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.03 seconds. BPR loss 1.33E+01. Sample per second: 13844\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.41 sec. Users per second: 1710\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.43 sec. Users per second: 1722\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2295247, PRECISION: 0.0834164, PRECISION_RECALL_MIN_DEN: 0.1050079, RECALL: 0.0677039, MAP: 0.0480973, MRR: 0.1906315, NDCG: 0.0673222, F1: 0.0747433, HIT_RATE: 0.8341639, ARHR: 0.2693694, NOVELTY: 0.0097712, AVERAGE_POPULARITY: 0.4980361, DIVERSITY_MEAN_INTER_LIST: 0.9030786, DIVERSITY_HERFINDAHL: 0.9903066, COVERAGE_ITEM: 0.9919483, COVERAGE_ITEM_CORRECT: 0.0593577, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4156101, DIVERSITY_GINI: 0.1369773, SHANNON_ENTROPY: 8.8658429, \n",
      "\n",
      "MF_BPR: Epoch 675 of 1500. Elapsed time 2.27 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.63 seconds. BPR loss 1.34E+01. Sample per second: 15034\n",
      "MF_BPR: Epoch 676 of 1500. Elapsed time 2.27 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.74 seconds. BPR loss 1.34E+01. Sample per second: 14680\n",
      "MF_BPR: Epoch 677 of 1500. Elapsed time 2.27 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.84 seconds. BPR loss 1.34E+01. Sample per second: 14382\n",
      "MF_BPR: Epoch 678 of 1500. Elapsed time 2.27 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 4.98 seconds. BPR loss 1.35E+01. Sample per second: 13986\n",
      "MF_BPR: Epoch 679 of 1500. Elapsed time 2.27 hour\n",
      "MF_BPR: Processed 69632 ( 99.63% ) in 5.24 seconds. BPR loss 1.34E+01. Sample per second: 13281\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.24 sec. Users per second: 1653\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.52 sec. Users per second: 1677\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2292565, PRECISION: 0.0834106, PRECISION_RECALL_MIN_DEN: 0.1049945, RECALL: 0.0677226, MAP: 0.0480774, MRR: 0.1904355, NDCG: 0.0673127, F1: 0.0747524, HIT_RATE: 0.8341064, ARHR: 0.2691759, NOVELTY: 0.0097715, AVERAGE_POPULARITY: 0.4973902, DIVERSITY_MEAN_INTER_LIST: 0.9036521, DIVERSITY_HERFINDAHL: 0.9903639, COVERAGE_ITEM: 0.9917611, COVERAGE_ITEM_CORRECT: 0.0593577, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4152094, DIVERSITY_GINI: 0.1367801, SHANNON_ENTROPY: 8.8694580, \n",
      "\n",
      "MF_BPR: Convergence reached! Terminating at epoch 680. Best value for 'MAP' at epoch 655 is 0.0481. Elapsed time 2.28 hour\n",
      "MF_BPR: Epoch 680 of 1500. Elapsed time 2.28 hour\n",
      "EvaluatorHoldout: Processed 47000 ( 67.49% ) in 30.31 sec. Users per second: 1551\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.97 sec. Users per second: 1584\n",
      "SearchBayesianSkopt: New best config found. Config 3: {'sgd_mode': 'adam', 'epochs': 655, 'num_factors': 199, 'batch_size': 256, 'positive_reg': 1e-05, 'negative_reg': 1e-05, 'learning_rate': 0.0001} - results: ROC_AUC: 0.2294309, PRECISION: 0.0834379, PRECISION_RECALL_MIN_DEN: 0.1049231, RECALL: 0.0675701, MAP: 0.0481448, MRR: 0.1911620, NDCG: 0.0672965, F1: 0.0746703, HIT_RATE: 0.8343793, ARHR: 0.2699014, NOVELTY: 0.0097714, AVERAGE_POPULARITY: 0.4995895, DIVERSITY_MEAN_INTER_LIST: 0.9020277, DIVERSITY_HERFINDAHL: 0.9902015, COVERAGE_ITEM: 0.9922292, COVERAGE_ITEM_CORRECT: 0.0592641, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4156530, DIVERSITY_GINI: 0.1381395, SHANNON_ENTROPY: 8.8560404, \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 50000 ( 71.65% ) in 30.51 sec. Users per second: 1639\n",
      "EvaluatorHoldout: Processed 69786 ( 100.00% ) in 42.05 sec. Users per second: 1660\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'sgd_mode': 'adam', 'epochs': 655, 'num_factors': 199, 'batch_size': 256, 'positive_reg': 1e-05, 'negative_reg': 1e-05, 'learning_rate': 0.0001} - results:\n",
      "CUTOFF: 10 - ROC_AUC: 0.2524456, PRECISION: 0.1037601, PRECISION_RECALL_MIN_DEN: 0.1197144, RECALL: 0.0670026, MAP: 0.0595471, MRR: 0.2229425, NDCG: 0.0721557, F1: 0.0814253, HIT_RATE: 1.0376007, ARHR: 0.3352805, NOVELTY: 0.0097716, AVERAGE_POPULARITY: 0.4995510, DIVERSITY_MEAN_INTER_LIST: 0.9020405, DIVERSITY_HERFINDAHL: 0.9902028, COVERAGE_ITEM: 0.9921356, COVERAGE_ITEM_CORRECT: 0.0661923, COVERAGE_USER: 0.9986834, COVERAGE_USER_CORRECT: 0.4559375, DIVERSITY_GINI: 0.1381223, SHANNON_ENTROPY: 8.8561728, \n",
      "\n",
      "\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 8307.0772\n",
      "Function value obtained: -0.0481\n",
      "Current minimum: -0.0481\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "MatrixFactorization_BPR_Cython_Recommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchBayesianSkopt: Testing config: {'sgd_mode': 'adam', 'epochs': 1500, 'num_factors': 200, 'batch_size': 1, 'positive_reg': 2.1264549316860416e-05, 'negative_reg': 0.00033305173444924584, 'learning_rate': 0.00010151223842175566}\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 3.99E-02. Sample per second: 13907\n",
      "MF_BPR: Epoch 1 of 1500. Elapsed time 4.54 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 3.95E-02. Sample per second: 16152\n",
      "MF_BPR: Epoch 2 of 1500. Elapsed time 8.84 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 3.96E-02. Sample per second: 15045\n",
      "MF_BPR: Epoch 3 of 1500. Elapsed time 13.16 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 3.98E-02. Sample per second: 14121\n",
      "MF_BPR: Epoch 4 of 1500. Elapsed time 17.47 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.38 seconds. BPR loss 3.98E-02. Sample per second: 12983\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.38 sec. Users per second: 1712\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.34 sec. Users per second: 1726\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0106951, PRECISION: 0.0021699, PRECISION_RECALL_MIN_DEN: 0.0023178, RECALL: 0.0008686, MAP: 0.0007174, MRR: 0.0064001, NDCG: 0.0008581, F1: 0.0012406, HIT_RATE: 0.0216989, ARHR: 0.0065317, NOVELTY: 0.0149547, AVERAGE_POPULARITY: 0.0245686, DIVERSITY_MEAN_INTER_LIST: 0.9987951, DIVERSITY_HERFINDAHL: 0.9998781, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0947477, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0207648, DIVERSITY_GINI: 0.7043079, SHANNON_ENTROPY: 13.1789418, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 5 of 1500. Elapsed time 1.04 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.11 seconds. BPR loss 3.98E-02. Sample per second: 13669\n",
      "MF_BPR: Epoch 6 of 1500. Elapsed time 1.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 3.97E-02. Sample per second: 15891\n",
      "MF_BPR: Epoch 7 of 1500. Elapsed time 1.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 3.98E-02. Sample per second: 14222\n",
      "MF_BPR: Epoch 8 of 1500. Elapsed time 1.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.24 seconds. BPR loss 3.97E-02. Sample per second: 13338\n",
      "MF_BPR: Epoch 9 of 1500. Elapsed time 1.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 3.97E-02. Sample per second: 15505\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.14 sec. Users per second: 1725\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.01 sec. Users per second: 1740\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0104646, PRECISION: 0.0021469, PRECISION_RECALL_MIN_DEN: 0.0022989, RECALL: 0.0008681, MAP: 0.0007081, MRR: 0.0063078, NDCG: 0.0008549, F1: 0.0012363, HIT_RATE: 0.0214691, ARHR: 0.0064363, NOVELTY: 0.0149586, AVERAGE_POPULARITY: 0.0244123, DIVERSITY_MEAN_INTER_LIST: 0.9987952, DIVERSITY_HERFINDAHL: 0.9998781, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0948413, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0205501, DIVERSITY_GINI: 0.7042437, SHANNON_ENTROPY: 13.1789419, \n",
      "\n",
      "MF_BPR: Epoch 10 of 1500. Elapsed time 2.07 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 4.02E-02. Sample per second: 14464\n",
      "MF_BPR: Epoch 11 of 1500. Elapsed time 2.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 4.01E-02. Sample per second: 13694\n",
      "MF_BPR: Epoch 12 of 1500. Elapsed time 2.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 3.97E-02. Sample per second: 16051\n",
      "MF_BPR: Epoch 13 of 1500. Elapsed time 2.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 4.00E-02. Sample per second: 15593\n",
      "MF_BPR: Epoch 14 of 1500. Elapsed time 2.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.58 seconds. BPR loss 3.95E-02. Sample per second: 15267\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 58000 ( 83.29% ) in 30.43 sec. Users per second: 1906\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 36.48 sec. Users per second: 1909\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0104945, PRECISION: 0.0021627, PRECISION_RECALL_MIN_DEN: 0.0023105, RECALL: 0.0008691, MAP: 0.0007132, MRR: 0.0063367, NDCG: 0.0008623, F1: 0.0012399, HIT_RATE: 0.0216271, ARHR: 0.0064632, NOVELTY: 0.0149567, AVERAGE_POPULARITY: 0.0244812, DIVERSITY_MEAN_INTER_LIST: 0.9987956, DIVERSITY_HERFINDAHL: 0.9998781, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0951222, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0207218, DIVERSITY_GINI: 0.7044340, SHANNON_ENTROPY: 13.1792137, \n",
      "\n",
      "MF_BPR: Epoch 15 of 1500. Elapsed time 3.03 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 4.01E-02. Sample per second: 16552\n",
      "MF_BPR: Epoch 16 of 1500. Elapsed time 3.10 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 4.01E-02. Sample per second: 16239\n",
      "MF_BPR: Epoch 17 of 1500. Elapsed time 3.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 4.03E-02. Sample per second: 15925\n",
      "MF_BPR: Epoch 18 of 1500. Elapsed time 3.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 4.02E-02. Sample per second: 15562\n",
      "MF_BPR: Epoch 19 of 1500. Elapsed time 3.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 4.02E-02. Sample per second: 14802\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.30 sec. Users per second: 1749\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.58 sec. Users per second: 1759\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0106168, PRECISION: 0.0021613, PRECISION_RECALL_MIN_DEN: 0.0023186, RECALL: 0.0008803, MAP: 0.0007284, MRR: 0.0064429, NDCG: 0.0008812, F1: 0.0012510, HIT_RATE: 0.0216127, ARHR: 0.0065722, NOVELTY: 0.0149527, AVERAGE_POPULARITY: 0.0246867, DIVERSITY_MEAN_INTER_LIST: 0.9987961, DIVERSITY_HERFINDAHL: 0.9998782, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0954031, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0207218, DIVERSITY_GINI: 0.7046857, SHANNON_ENTROPY: 13.1796183, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 20 of 1500. Elapsed time 4.03 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 4.01E-02. Sample per second: 14871\n",
      "MF_BPR: Epoch 21 of 1500. Elapsed time 4.10 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 4.02E-02. Sample per second: 13804\n",
      "MF_BPR: Epoch 22 of 1500. Elapsed time 4.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 4.02E-02. Sample per second: 16111\n",
      "MF_BPR: Epoch 23 of 1500. Elapsed time 4.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 4.02E-02. Sample per second: 15346\n",
      "MF_BPR: Epoch 24 of 1500. Elapsed time 4.32 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 4.04E-02. Sample per second: 14644\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.07 sec. Users per second: 1762\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.38 sec. Users per second: 1768\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0107743, PRECISION: 0.0021814, PRECISION_RECALL_MIN_DEN: 0.0023425, RECALL: 0.0009010, MAP: 0.0007329, MRR: 0.0064692, NDCG: 0.0008986, F1: 0.0012753, HIT_RATE: 0.0218137, ARHR: 0.0066051, NOVELTY: 0.0149433, AVERAGE_POPULARITY: 0.0250697, DIVERSITY_MEAN_INTER_LIST: 0.9987963, DIVERSITY_HERFINDAHL: 0.9998782, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0950286, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0208649, DIVERSITY_GINI: 0.7047721, SHANNON_ENTROPY: 13.1797672, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 25 of 1500. Elapsed time 5.05 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 4.05E-02. Sample per second: 14935\n",
      "MF_BPR: Epoch 26 of 1500. Elapsed time 5.12 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 4.06E-02. Sample per second: 13639\n",
      "MF_BPR: Epoch 27 of 1500. Elapsed time 5.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 4.07E-02. Sample per second: 15431\n",
      "MF_BPR: Epoch 28 of 1500. Elapsed time 5.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 4.06E-02. Sample per second: 14506\n",
      "MF_BPR: Epoch 29 of 1500. Elapsed time 5.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.25 seconds. BPR loss 4.06E-02. Sample per second: 13322\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.09 sec. Users per second: 1662\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.90 sec. Users per second: 1662\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0111499, PRECISION: 0.0022546, PRECISION_RECALL_MIN_DEN: 0.0024258, RECALL: 0.0009488, MAP: 0.0007634, MRR: 0.0067307, NDCG: 0.0009464, F1: 0.0013356, HIT_RATE: 0.0225461, ARHR: 0.0068748, NOVELTY: 0.0149315, AVERAGE_POPULARITY: 0.0256764, DIVERSITY_MEAN_INTER_LIST: 0.9987971, DIVERSITY_HERFINDAHL: 0.9998783, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0964329, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0215375, DIVERSITY_GINI: 0.7052241, SHANNON_ENTROPY: 13.1803939, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 30 of 1500. Elapsed time 6.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 4.07E-02. Sample per second: 14666\n",
      "MF_BPR: Epoch 31 of 1500. Elapsed time 6.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 4.08E-02. Sample per second: 13638\n",
      "MF_BPR: Epoch 32 of 1500. Elapsed time 6.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 4.10E-02. Sample per second: 15404\n",
      "MF_BPR: Epoch 33 of 1500. Elapsed time 6.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 4.09E-02. Sample per second: 13965\n",
      "MF_BPR: Epoch 34 of 1500. Elapsed time 6.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.43 seconds. BPR loss 4.10E-02. Sample per second: 15769\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.34 sec. Users per second: 1747\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.61 sec. Users per second: 1758\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0115385, PRECISION: 0.0023192, PRECISION_RECALL_MIN_DEN: 0.0025067, RECALL: 0.0010039, MAP: 0.0007965, MRR: 0.0069799, NDCG: 0.0009965, F1: 0.0014013, HIT_RATE: 0.0231924, ARHR: 0.0071355, NOVELTY: 0.0149157, AVERAGE_POPULARITY: 0.0264892, DIVERSITY_MEAN_INTER_LIST: 0.9987965, DIVERSITY_HERFINDAHL: 0.9998782, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0958712, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0221100, DIVERSITY_GINI: 0.7049727, SHANNON_ENTROPY: 13.1800049, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 35 of 1500. Elapsed time 7.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 4.12E-02. Sample per second: 15920\n",
      "MF_BPR: Epoch 36 of 1500. Elapsed time 7.22 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 4.10E-02. Sample per second: 15084\n",
      "MF_BPR: Epoch 37 of 1500. Elapsed time 7.29 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 4.17E-02. Sample per second: 14397\n",
      "MF_BPR: Epoch 38 of 1500. Elapsed time 7.36 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 4.15E-02. Sample per second: 13720\n",
      "MF_BPR: Epoch 39 of 1500. Elapsed time 7.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 4.15E-02. Sample per second: 16189\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.28 sec. Users per second: 1751\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.76 sec. Users per second: 1751\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0121238, PRECISION: 0.0024312, PRECISION_RECALL_MIN_DEN: 0.0026399, RECALL: 0.0010853, MAP: 0.0008530, MRR: 0.0074483, NDCG: 0.0010893, F1: 0.0015007, HIT_RATE: 0.0243125, ARHR: 0.0076002, NOVELTY: 0.0148978, AVERAGE_POPULARITY: 0.0275818, DIVERSITY_MEAN_INTER_LIST: 0.9987954, DIVERSITY_HERFINDAHL: 0.9998781, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0957775, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0232405, DIVERSITY_GINI: 0.7045926, SHANNON_ENTROPY: 13.1794245, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 40 of 1500. Elapsed time 8.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 4.14E-02. Sample per second: 15810\n",
      "MF_BPR: Epoch 41 of 1500. Elapsed time 8.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 4.15E-02. Sample per second: 14952\n",
      "MF_BPR: Epoch 42 of 1500. Elapsed time 8.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.11 seconds. BPR loss 4.19E-02. Sample per second: 13685\n",
      "MF_BPR: Epoch 43 of 1500. Elapsed time 8.38 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 4.17E-02. Sample per second: 15906\n",
      "MF_BPR: Epoch 44 of 1500. Elapsed time 8.45 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 4.23E-02. Sample per second: 14731\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.08 sec. Users per second: 1729\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.47 sec. Users per second: 1679\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0128999, PRECISION: 0.0026151, PRECISION_RECALL_MIN_DEN: 0.0028753, RECALL: 0.0012331, MAP: 0.0009214, MRR: 0.0079473, NDCG: 0.0012160, F1: 0.0016759, HIT_RATE: 0.0261506, ARHR: 0.0081148, NOVELTY: 0.0148732, AVERAGE_POPULARITY: 0.0290854, DIVERSITY_MEAN_INTER_LIST: 0.9987929, DIVERSITY_HERFINDAHL: 0.9998779, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0966202, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0249864, DIVERSITY_GINI: 0.7038786, SHANNON_ENTROPY: 13.1781763, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 45 of 1500. Elapsed time 9.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 4.24E-02. Sample per second: 14897\n",
      "MF_BPR: Epoch 46 of 1500. Elapsed time 9.29 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 4.24E-02. Sample per second: 13830\n",
      "MF_BPR: Epoch 47 of 1500. Elapsed time 9.36 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 4.27E-02. Sample per second: 15872\n",
      "MF_BPR: Epoch 48 of 1500. Elapsed time 9.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 4.29E-02. Sample per second: 14748\n",
      "MF_BPR: Epoch 49 of 1500. Elapsed time 9.50 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 4.29E-02. Sample per second: 13806\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.08 sec. Users per second: 1695\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.51 sec. Users per second: 1719\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0141520, PRECISION: 0.0028333, PRECISION_RECALL_MIN_DEN: 0.0031711, RECALL: 0.0014611, MAP: 0.0010248, MRR: 0.0087173, NDCG: 0.0014167, F1: 0.0019280, HIT_RATE: 0.0283335, ARHR: 0.0088995, NOVELTY: 0.0148446, AVERAGE_POPULARITY: 0.0310950, DIVERSITY_MEAN_INTER_LIST: 0.9987858, DIVERSITY_HERFINDAHL: 0.9998771, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0981182, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0270614, DIVERSITY_GINI: 0.7021325, SHANNON_ENTROPY: 13.1750023, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 50 of 1500. Elapsed time 10.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 4.26E-02. Sample per second: 13800\n",
      "MF_BPR: Epoch 51 of 1500. Elapsed time 10.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 4.35E-02. Sample per second: 16057\n",
      "MF_BPR: Epoch 52 of 1500. Elapsed time 10.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 4.34E-02. Sample per second: 15116\n",
      "MF_BPR: Epoch 53 of 1500. Elapsed time 10.47 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 4.33E-02. Sample per second: 14205\n",
      "MF_BPR: Epoch 54 of 1500. Elapsed time 10.54 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.23 seconds. BPR loss 4.41E-02. Sample per second: 13361\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.31 sec. Users per second: 1749\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.31 sec. Users per second: 1727\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0159476, PRECISION: 0.0031636, PRECISION_RECALL_MIN_DEN: 0.0035973, RECALL: 0.0017539, MAP: 0.0011956, MRR: 0.0099415, NDCG: 0.0017084, F1: 0.0022567, HIT_RATE: 0.0316364, ARHR: 0.0101678, NOVELTY: 0.0148085, AVERAGE_POPULARITY: 0.0337421, DIVERSITY_MEAN_INTER_LIST: 0.9987710, DIVERSITY_HERFINDAHL: 0.9998757, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0974628, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0301382, DIVERSITY_GINI: 0.6995677, SHANNON_ENTROPY: 13.1694717, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 55 of 1500. Elapsed time 11.29 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 4.40E-02. Sample per second: 14106\n",
      "MF_BPR: Epoch 56 of 1500. Elapsed time 11.36 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.32 seconds. BPR loss 4.44E-02. Sample per second: 13125\n",
      "MF_BPR: Epoch 57 of 1500. Elapsed time 11.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 4.43E-02. Sample per second: 15009\n",
      "MF_BPR: Epoch 58 of 1500. Elapsed time 11.50 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 4.51E-02. Sample per second: 13720\n",
      "MF_BPR: Epoch 59 of 1500. Elapsed time 11.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 4.46E-02. Sample per second: 15609\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.54 sec. Users per second: 1670\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.46 sec. Users per second: 1679\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0185004, PRECISION: 0.0036821, PRECISION_RECALL_MIN_DEN: 0.0042807, RECALL: 0.0022518, MAP: 0.0014403, MRR: 0.0115994, NDCG: 0.0021640, F1: 0.0027946, HIT_RATE: 0.0368206, ARHR: 0.0119452, NOVELTY: 0.0147595, AVERAGE_POPULARITY: 0.0374679, DIVERSITY_MEAN_INTER_LIST: 0.9987405, DIVERSITY_HERFINDAHL: 0.9998726, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0983054, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0347033, DIVERSITY_GINI: 0.6960248, SHANNON_ENTROPY: 13.1602152, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 60 of 1500. Elapsed time 12.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 4.53E-02. Sample per second: 15342\n",
      "MF_BPR: Epoch 61 of 1500. Elapsed time 12.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 4.57E-02. Sample per second: 14115\n",
      "MF_BPR: Epoch 62 of 1500. Elapsed time 12.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.33 seconds. BPR loss 4.60E-02. Sample per second: 13103\n",
      "MF_BPR: Epoch 63 of 1500. Elapsed time 12.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 4.64E-02. Sample per second: 14987\n",
      "MF_BPR: Epoch 64 of 1500. Elapsed time 12.64 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 4.67E-02. Sample per second: 14010\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.03 sec. Users per second: 1665\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.88 sec. Users per second: 1663\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0224444, PRECISION: 0.0044044, PRECISION_RECALL_MIN_DEN: 0.0053076, RECALL: 0.0030303, MAP: 0.0018714, MRR: 0.0143825, NDCG: 0.0029034, F1: 0.0035904, HIT_RATE: 0.0440439, ARHR: 0.0149232, NOVELTY: 0.0146926, AVERAGE_POPULARITY: 0.0427173, DIVERSITY_MEAN_INTER_LIST: 0.9986741, DIVERSITY_HERFINDAHL: 0.9998660, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0998970, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0408426, DIVERSITY_GINI: 0.6905621, SHANNON_ENTROPY: 13.1435789, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 65 of 1500. Elapsed time 13.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.48 seconds. BPR loss 4.67E-02. Sample per second: 12747\n",
      "MF_BPR: Epoch 66 of 1500. Elapsed time 13.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 4.75E-02. Sample per second: 14520\n",
      "MF_BPR: Epoch 67 of 1500. Elapsed time 13.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.18 seconds. BPR loss 4.82E-02. Sample per second: 13493\n",
      "MF_BPR: Epoch 68 of 1500. Elapsed time 13.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 4.83E-02. Sample per second: 15388\n",
      "MF_BPR: Epoch 69 of 1500. Elapsed time 13.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 4.90E-02. Sample per second: 14171\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.35 sec. Users per second: 1713\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.55 sec. Users per second: 1717\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0275310, PRECISION: 0.0053637, PRECISION_RECALL_MIN_DEN: 0.0066513, RECALL: 0.0040622, MAP: 0.0025168, MRR: 0.0181471, NDCG: 0.0039345, F1: 0.0046231, HIT_RATE: 0.0536368, ARHR: 0.0190968, NOVELTY: 0.0146096, AVERAGE_POPULARITY: 0.0496415, DIVERSITY_MEAN_INTER_LIST: 0.9985416, DIVERSITY_HERFINDAHL: 0.9998527, COVERAGE_ITEM: 0.9999064, COVERAGE_ITEM_CORRECT: 0.0998034, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0484559, DIVERSITY_GINI: 0.6828072, SHANNON_ENTROPY: 13.1165423, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 70 of 1500. Elapsed time 14.45 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 4.98E-02. Sample per second: 14180\n",
      "MF_BPR: Epoch 71 of 1500. Elapsed time 14.52 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.28 seconds. BPR loss 4.96E-02. Sample per second: 13233\n",
      "MF_BPR: Epoch 72 of 1500. Elapsed time 14.60 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 5.01E-02. Sample per second: 15140\n",
      "MF_BPR: Epoch 73 of 1500. Elapsed time 14.67 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 5.04E-02. Sample per second: 14049\n",
      "MF_BPR: Epoch 74 of 1500. Elapsed time 14.74 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.33 seconds. BPR loss 5.10E-02. Sample per second: 13120\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.13 sec. Users per second: 1659\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.86 sec. Users per second: 1663\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0341058, PRECISION: 0.0066547, PRECISION_RECALL_MIN_DEN: 0.0085828, RECALL: 0.0056043, MAP: 0.0034402, MRR: 0.0231276, NDCG: 0.0054187, F1: 0.0060845, HIT_RATE: 0.0665470, ARHR: 0.0247276, NOVELTY: 0.0145016, AVERAGE_POPULARITY: 0.0589343, DIVERSITY_MEAN_INTER_LIST: 0.9982808, DIVERSITY_HERFINDAHL: 0.9998266, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0998970, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0585163, DIVERSITY_GINI: 0.6721668, SHANNON_ENTROPY: 13.0734853, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 75 of 1500. Elapsed time 15.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 5.20E-02. Sample per second: 15121\n",
      "MF_BPR: Epoch 76 of 1500. Elapsed time 15.59 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 5.34E-02. Sample per second: 14070\n",
      "MF_BPR: Epoch 77 of 1500. Elapsed time 15.66 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.43 seconds. BPR loss 5.35E-02. Sample per second: 12877\n",
      "MF_BPR: Epoch 78 of 1500. Elapsed time 15.73 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 5.44E-02. Sample per second: 14439\n",
      "MF_BPR: Epoch 79 of 1500. Elapsed time 15.81 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.18 seconds. BPR loss 5.44E-02. Sample per second: 13482\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.38 sec. Users per second: 1613\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.33 sec. Users per second: 1607\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0422425, PRECISION: 0.0084354, PRECISION_RECALL_MIN_DEN: 0.0112504, RECALL: 0.0077808, MAP: 0.0047416, MRR: 0.0293135, NDCG: 0.0074404, F1: 0.0080949, HIT_RATE: 0.0843541, ARHR: 0.0320263, NOVELTY: 0.0143628, AVERAGE_POPULARITY: 0.0709751, DIVERSITY_MEAN_INTER_LIST: 0.9978004, DIVERSITY_HERFINDAHL: 0.9997786, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0995225, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0707519, DIVERSITY_GINI: 0.6579631, SHANNON_ENTROPY: 13.0087452, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 80 of 1500. Elapsed time 16.60 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.96 seconds. BPR loss 5.59E-02. Sample per second: 14075\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 81 of 1500. Elapsed time 16.67 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.28 seconds. BPR loss 5.65E-02. Sample per second: 13229\n",
      "MF_BPR: Epoch 82 of 1500. Elapsed time 16.75 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 5.79E-02. Sample per second: 15119\n",
      "MF_BPR: Epoch 83 of 1500. Elapsed time 16.82 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 5.82E-02. Sample per second: 14046\n",
      "MF_BPR: Epoch 84 of 1500. Elapsed time 16.89 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.34 seconds. BPR loss 5.96E-02. Sample per second: 13098\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.57 sec. Users per second: 1734\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.18 sec. Users per second: 1733\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0518910, PRECISION: 0.0108178, PRECISION_RECALL_MIN_DEN: 0.0148525, RECALL: 0.0106926, MAP: 0.0064567, MRR: 0.0369599, NDCG: 0.0100396, F1: 0.0107548, HIT_RATE: 0.1081784, ARHR: 0.0414162, NOVELTY: 0.0141944, AVERAGE_POPULARITY: 0.0863926, DIVERSITY_MEAN_INTER_LIST: 0.9969343, DIVERSITY_HERFINDAHL: 0.9996920, COVERAGE_ITEM: 0.9999064, COVERAGE_ITEM_CORRECT: 0.0986799, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.0861931, DIVERSITY_GINI: 0.6397960, SHANNON_ENTROPY: 12.9145231, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 85 of 1500. Elapsed time 17.64 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 6.11E-02. Sample per second: 14123\n",
      "MF_BPR: Epoch 86 of 1500. Elapsed time 17.71 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.20 seconds. BPR loss 6.21E-02. Sample per second: 13440\n",
      "MF_BPR: Epoch 87 of 1500. Elapsed time 17.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 6.20E-02. Sample per second: 15493\n",
      "MF_BPR: Epoch 88 of 1500. Elapsed time 17.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 6.43E-02. Sample per second: 13636\n",
      "MF_BPR: Epoch 89 of 1500. Elapsed time 17.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 6.49E-02. Sample per second: 15439\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.03 sec. Users per second: 1698\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.44 sec. Users per second: 1681\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0625524, PRECISION: 0.0138149, PRECISION_RECALL_MIN_DEN: 0.0193001, RECALL: 0.0142615, MAP: 0.0085086, MRR: 0.0454128, NDCG: 0.0131983, F1: 0.0140347, HIT_RATE: 0.1381489, ARHR: 0.0522824, NOVELTY: 0.0139951, AVERAGE_POPULARITY: 0.1043846, DIVERSITY_MEAN_INTER_LIST: 0.9955814, DIVERSITY_HERFINDAHL: 0.9995567, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0989608, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1041530, DIVERSITY_GINI: 0.6189377, SHANNON_ENTROPY: 12.7931913, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 90 of 1500. Elapsed time 18.69 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 6.75E-02. Sample per second: 16108\n",
      "MF_BPR: Epoch 91 of 1500. Elapsed time 18.76 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 6.85E-02. Sample per second: 15297\n",
      "MF_BPR: Epoch 92 of 1500. Elapsed time 18.83 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 6.95E-02. Sample per second: 14606\n",
      "MF_BPR: Epoch 93 of 1500. Elapsed time 18.91 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 7.09E-02. Sample per second: 13941\n",
      "MF_BPR: Epoch 94 of 1500. Elapsed time 18.98 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.26 seconds. BPR loss 7.42E-02. Sample per second: 16392\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.36 sec. Users per second: 1680\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.78 sec. Users per second: 1708\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0746577, PRECISION: 0.0170762, PRECISION_RECALL_MIN_DEN: 0.0239441, RECALL: 0.0179438, MAP: 0.0107913, MRR: 0.0547903, NDCG: 0.0165880, F1: 0.0174993, HIT_RATE: 0.1707618, ARHR: 0.0645212, NOVELTY: 0.0137753, AVERAGE_POPULARITY: 0.1247920, DIVERSITY_MEAN_INTER_LIST: 0.9935852, DIVERSITY_HERFINDAHL: 0.9993571, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0978373, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1227139, DIVERSITY_GINI: 0.5958108, SHANNON_ENTROPY: 12.6436301, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 95 of 1500. Elapsed time 19.73 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 7.61E-02. Sample per second: 16049\n",
      "MF_BPR: Epoch 96 of 1500. Elapsed time 19.80 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 7.76E-02. Sample per second: 15332\n",
      "MF_BPR: Epoch 97 of 1500. Elapsed time 19.87 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 7.91E-02. Sample per second: 14694\n",
      "MF_BPR: Epoch 98 of 1500. Elapsed time 19.94 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 8.13E-02. Sample per second: 14174\n",
      "MF_BPR: Epoch 99 of 1500. Elapsed time 20.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.19 seconds. BPR loss 8.37E-02. Sample per second: 13471\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.38 sec. Users per second: 1745\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.82 sec. Users per second: 1749\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0863599, PRECISION: 0.0207137, PRECISION_RECALL_MIN_DEN: 0.0291391, RECALL: 0.0220103, MAP: 0.0132150, MRR: 0.0642217, NDCG: 0.0201707, F1: 0.0213423, HIT_RATE: 0.2071372, ARHR: 0.0773038, NOVELTY: 0.0135309, AVERAGE_POPULARITY: 0.1475101, DIVERSITY_MEAN_INTER_LIST: 0.9907678, DIVERSITY_HERFINDAHL: 0.9990754, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0964329, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1417757, DIVERSITY_GINI: 0.5695004, SHANNON_ENTROPY: 12.4626062, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 100 of 1500. Elapsed time 20.74 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 8.58E-02. Sample per second: 15547\n",
      "MF_BPR: Epoch 101 of 1500. Elapsed time 20.82 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 8.75E-02. Sample per second: 14485\n",
      "MF_BPR: Epoch 102 of 1500. Elapsed time 20.89 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 9.05E-02. Sample per second: 13722\n",
      "MF_BPR: Epoch 103 of 1500. Elapsed time 20.96 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 9.50E-02. Sample per second: 16082\n",
      "MF_BPR: Epoch 104 of 1500. Elapsed time 21.03 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.58 seconds. BPR loss 9.67E-02. Sample per second: 15248\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.51 sec. Users per second: 1737\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.38 sec. Users per second: 1725\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.0973445, PRECISION: 0.0244848, PRECISION_RECALL_MIN_DEN: 0.0344293, RECALL: 0.0261285, MAP: 0.0156615, MRR: 0.0734371, NDCG: 0.0236307, F1: 0.0252800, HIT_RATE: 0.2448481, ARHR: 0.0901127, NOVELTY: 0.0132821, AVERAGE_POPULARITY: 0.1708765, DIVERSITY_MEAN_INTER_LIST: 0.9872100, DIVERSITY_HERFINDAHL: 0.9987196, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0961520, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1608947, DIVERSITY_GINI: 0.5426388, SHANNON_ENTROPY: 12.2644845, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 105 of 1500. Elapsed time 21.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 1.01E-01. Sample per second: 15997\n",
      "MF_BPR: Epoch 106 of 1500. Elapsed time 21.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 1.03E-01. Sample per second: 15044\n",
      "MF_BPR: Epoch 107 of 1500. Elapsed time 21.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.06E-01. Sample per second: 14233\n",
      "MF_BPR: Epoch 108 of 1500. Elapsed time 21.99 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.18 seconds. BPR loss 1.11E-01. Sample per second: 13498\n",
      "MF_BPR: Epoch 109 of 1500. Elapsed time 22.06 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 1.14E-01. Sample per second: 15692\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.36 sec. Users per second: 1713\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.86 sec. Users per second: 1704\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1079920, PRECISION: 0.0283837, PRECISION_RECALL_MIN_DEN: 0.0395654, RECALL: 0.0298980, MAP: 0.0180173, MRR: 0.0823802, NDCG: 0.0269219, F1: 0.0291212, HIT_RATE: 0.2838372, ARHR: 0.1029098, NOVELTY: 0.0130280, AVERAGE_POPULARITY: 0.1949107, DIVERSITY_MEAN_INTER_LIST: 0.9828493, DIVERSITY_HERFINDAHL: 0.9982835, COVERAGE_ITEM: 0.9998128, COVERAGE_ITEM_CORRECT: 0.0938114, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1797991, DIVERSITY_GINI: 0.5152450, SHANNON_ENTROPY: 12.0493562, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 110 of 1500. Elapsed time 22.82 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.19E-01. Sample per second: 14416\n",
      "MF_BPR: Epoch 111 of 1500. Elapsed time 22.89 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 1.22E-01. Sample per second: 13563\n",
      "MF_BPR: Epoch 112 of 1500. Elapsed time 22.96 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 1.26E-01. Sample per second: 14370\n",
      "MF_BPR: Epoch 113 of 1500. Elapsed time 23.04 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 1.30E-01. Sample per second: 13538\n",
      "MF_BPR: Epoch 114 of 1500. Elapsed time 23.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.36E-01. Sample per second: 15527\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.11 sec. Users per second: 1661\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.61 sec. Users per second: 1674\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1179340, PRECISION: 0.0322295, PRECISION_RECALL_MIN_DEN: 0.0446657, RECALL: 0.0335857, MAP: 0.0203629, MRR: 0.0909526, NDCG: 0.0301160, F1: 0.0328936, HIT_RATE: 0.3222948, ARHR: 0.1153750, NOVELTY: 0.0127775, AVERAGE_POPULARITY: 0.2189035, DIVERSITY_MEAN_INTER_LIST: 0.9777834, DIVERSITY_HERFINDAHL: 0.9977769, COVERAGE_ITEM: 0.9997191, COVERAGE_ITEM_CORRECT: 0.0927816, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.1975300, DIVERSITY_GINI: 0.4883192, SHANNON_ENTROPY: 11.8250816, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 115 of 1500. Elapsed time 23.88 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 1.41E-01. Sample per second: 15337\n",
      "MF_BPR: Epoch 116 of 1500. Elapsed time 23.95 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.19 seconds. BPR loss 1.44E-01. Sample per second: 13457\n",
      "MF_BPR: Epoch 117 of 1500. Elapsed time 24.03 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 1.50E-01. Sample per second: 15036\n",
      "MF_BPR: Epoch 118 of 1500. Elapsed time 24.10 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 1.54E-01. Sample per second: 13915\n",
      "MF_BPR: Epoch 119 of 1500. Elapsed time 24.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 1.61E-01. Sample per second: 16080\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.38 sec. Users per second: 1679\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.17 sec. Users per second: 1692\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1269009, PRECISION: 0.0359460, PRECISION_RECALL_MIN_DEN: 0.0493423, RECALL: 0.0368036, MAP: 0.0225059, MRR: 0.0987113, NDCG: 0.0329942, F1: 0.0363698, HIT_RATE: 0.3594600, ARHR: 0.1268568, NOVELTY: 0.0125404, AVERAGE_POPULARITY: 0.2418111, DIVERSITY_MEAN_INTER_LIST: 0.9723634, DIVERSITY_HERFINDAHL: 0.9972349, COVERAGE_ITEM: 0.9997191, COVERAGE_ITEM_CORRECT: 0.0910963, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2146169, DIVERSITY_GINI: 0.4626945, SHANNON_ENTROPY: 11.6044360, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 120 of 1500. Elapsed time 24.94 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 1.68E-01. Sample per second: 14151\n",
      "MF_BPR: Epoch 121 of 1500. Elapsed time 25.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.25 seconds. BPR loss 1.74E-01. Sample per second: 13307\n",
      "MF_BPR: Epoch 122 of 1500. Elapsed time 25.08 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 1.81E-01. Sample per second: 14718\n",
      "MF_BPR: Epoch 123 of 1500. Elapsed time 25.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 1.89E-01. Sample per second: 13866\n",
      "MF_BPR: Epoch 124 of 1500. Elapsed time 25.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.92E-01. Sample per second: 16117\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.50 sec. Users per second: 1738\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.16 sec. Users per second: 1734\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1354361, PRECISION: 0.0394083, PRECISION_RECALL_MIN_DEN: 0.0536847, RECALL: 0.0397274, MAP: 0.0245332, MRR: 0.1060982, NDCG: 0.0356947, F1: 0.0395672, HIT_RATE: 0.3940834, ARHR: 0.1378361, NOVELTY: 0.0123114, AVERAGE_POPULARITY: 0.2640380, DIVERSITY_MEAN_INTER_LIST: 0.9665127, DIVERSITY_HERFINDAHL: 0.9966499, COVERAGE_ITEM: 0.9997191, COVERAGE_ITEM_CORRECT: 0.0892239, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2304159, DIVERSITY_GINI: 0.4375287, SHANNON_ENTROPY: 11.3830184, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 125 of 1500. Elapsed time 25.97 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 1.99E-01. Sample per second: 14286\n",
      "MF_BPR: Epoch 126 of 1500. Elapsed time 26.04 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.25 seconds. BPR loss 2.12E-01. Sample per second: 13307\n",
      "MF_BPR: Epoch 127 of 1500. Elapsed time 26.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 2.17E-01. Sample per second: 14974\n",
      "MF_BPR: Epoch 128 of 1500. Elapsed time 26.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 2.27E-01. Sample per second: 13735\n",
      "MF_BPR: Epoch 129 of 1500. Elapsed time 26.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 2.28E-01. Sample per second: 14719\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.56 sec. Users per second: 1636\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.82 sec. Users per second: 1665\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1432497, PRECISION: 0.0426237, PRECISION_RECALL_MIN_DEN: 0.0575887, RECALL: 0.0422502, MAP: 0.0263826, MRR: 0.1129265, NDCG: 0.0381370, F1: 0.0424361, HIT_RATE: 0.4262368, ARHR: 0.1479973, NOVELTY: 0.0120953, AVERAGE_POPULARITY: 0.2849702, DIVERSITY_MEAN_INTER_LIST: 0.9605012, DIVERSITY_HERFINDAHL: 0.9960487, COVERAGE_ITEM: 0.9996255, COVERAGE_ITEM_CORRECT: 0.0873514, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2445977, DIVERSITY_GINI: 0.4138180, SHANNON_ENTROPY: 11.1680395, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 130 of 1500. Elapsed time 27.04 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 2.38E-01. Sample per second: 13710\n",
      "MF_BPR: Epoch 131 of 1500. Elapsed time 27.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 2.49E-01. Sample per second: 15668\n",
      "MF_BPR: Epoch 132 of 1500. Elapsed time 27.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 2.63E-01. Sample per second: 14000\n",
      "MF_BPR: Epoch 133 of 1500. Elapsed time 27.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.43 seconds. BPR loss 2.62E-01. Sample per second: 12868\n",
      "MF_BPR: Epoch 134 of 1500. Elapsed time 27.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 2.85E-01. Sample per second: 14337\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.37% ) in 30.53 sec. Users per second: 1605\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.64 sec. Users per second: 1596\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1507668, PRECISION: 0.0457213, PRECISION_RECALL_MIN_DEN: 0.0613261, RECALL: 0.0446070, MAP: 0.0281633, MRR: 0.1194377, NDCG: 0.0404389, F1: 0.0451573, HIT_RATE: 0.4572126, ARHR: 0.1577131, NOVELTY: 0.0118909, AVERAGE_POPULARITY: 0.3049461, DIVERSITY_MEAN_INTER_LIST: 0.9542993, DIVERSITY_HERFINDAHL: 0.9954286, COVERAGE_ITEM: 0.9995319, COVERAGE_ITEM_CORRECT: 0.0854789, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2584934, DIVERSITY_GINI: 0.3914348, SHANNON_ENTROPY: 10.9595153, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 135 of 1500. Elapsed time 28.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 2.91E-01. Sample per second: 14721\n",
      "MF_BPR: Epoch 136 of 1500. Elapsed time 28.20 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 2.96E-01. Sample per second: 14405\n",
      "MF_BPR: Epoch 137 of 1500. Elapsed time 28.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 3.15E-01. Sample per second: 13967\n",
      "MF_BPR: Epoch 138 of 1500. Elapsed time 28.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.08 seconds. BPR loss 3.18E-01. Sample per second: 17111\n",
      "MF_BPR: Epoch 139 of 1500. Elapsed time 28.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.19 seconds. BPR loss 3.31E-01. Sample per second: 16686\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.36 sec. Users per second: 1845\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.75 sec. Users per second: 1845\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1575105, PRECISION: 0.0484541, PRECISION_RECALL_MIN_DEN: 0.0646150, RECALL: 0.0465788, MAP: 0.0297380, MRR: 0.1254415, NDCG: 0.0424833, F1: 0.0474980, HIT_RATE: 0.4845408, ARHR: 0.1665306, NOVELTY: 0.0117003, AVERAGE_POPULARITY: 0.3236594, DIVERSITY_MEAN_INTER_LIST: 0.9481909, DIVERSITY_HERFINDAHL: 0.9948177, COVERAGE_ITEM: 0.9994383, COVERAGE_ITEM_CORRECT: 0.0829510, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2706145, DIVERSITY_GINI: 0.3706576, SHANNON_ENTROPY: 10.7607230, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 140 of 1500. Elapsed time 29.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.17 seconds. BPR loss 3.49E-01. Sample per second: 16739\n",
      "MF_BPR: Epoch 141 of 1500. Elapsed time 29.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 3.56E-01. Sample per second: 16177\n",
      "MF_BPR: Epoch 142 of 1500. Elapsed time 29.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 3.68E-01. Sample per second: 15645\n",
      "MF_BPR: Epoch 143 of 1500. Elapsed time 29.32 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 3.88E-01. Sample per second: 15295\n",
      "MF_BPR: Epoch 144 of 1500. Elapsed time 29.38 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 3.94E-01. Sample per second: 14840\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.40 sec. Users per second: 1809\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.53 sec. Users per second: 1807\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1632763, PRECISION: 0.0510160, PRECISION_RECALL_MIN_DEN: 0.0677679, RECALL: 0.0485630, MAP: 0.0311701, MRR: 0.1308997, NDCG: 0.0443168, F1: 0.0497593, HIT_RATE: 0.5101601, ARHR: 0.1745680, NOVELTY: 0.0115251, AVERAGE_POPULARITY: 0.3412156, DIVERSITY_MEAN_INTER_LIST: 0.9420717, DIVERSITY_HERFINDAHL: 0.9942058, COVERAGE_ITEM: 0.9994383, COVERAGE_ITEM_CORRECT: 0.0811722, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2819199, DIVERSITY_GINI: 0.3515742, SHANNON_ENTROPY: 10.5721019, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 145 of 1500. Elapsed time 30.10 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 4.06E-01. Sample per second: 15240\n",
      "MF_BPR: Epoch 146 of 1500. Elapsed time 30.17 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 4.25E-01. Sample per second: 14108\n",
      "MF_BPR: Epoch 147 of 1500. Elapsed time 30.24 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.30 seconds. BPR loss 4.35E-01. Sample per second: 13175\n",
      "MF_BPR: Epoch 148 of 1500. Elapsed time 30.31 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 4.51E-01. Sample per second: 15034\n",
      "MF_BPR: Epoch 149 of 1500. Elapsed time 30.39 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 4.59E-01. Sample per second: 14193\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 47000 ( 67.49% ) in 30.11 sec. Users per second: 1561\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 43.88 sec. Users per second: 1587\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1686000, PRECISION: 0.0534200, PRECISION_RECALL_MIN_DEN: 0.0706281, RECALL: 0.0502724, MAP: 0.0324764, MRR: 0.1359826, NDCG: 0.0459543, F1: 0.0517984, HIT_RATE: 0.5341998, ARHR: 0.1820144, NOVELTY: 0.0113633, AVERAGE_POPULARITY: 0.3574447, DIVERSITY_MEAN_INTER_LIST: 0.9361213, DIVERSITY_HERFINDAHL: 0.9936108, COVERAGE_ITEM: 0.9992510, COVERAGE_ITEM_CORRECT: 0.0786443, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.2931824, DIVERSITY_GINI: 0.3339017, SHANNON_ENTROPY: 10.3945802, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 150 of 1500. Elapsed time 31.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.35 seconds. BPR loss 4.81E-01. Sample per second: 13056\n",
      "MF_BPR: Epoch 151 of 1500. Elapsed time 31.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 4.86E-01. Sample per second: 14700\n",
      "MF_BPR: Epoch 152 of 1500. Elapsed time 31.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 5.08E-01. Sample per second: 13658\n",
      "MF_BPR: Epoch 153 of 1500. Elapsed time 31.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 5.26E-01. Sample per second: 15746\n",
      "MF_BPR: Epoch 154 of 1500. Elapsed time 31.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 5.34E-01. Sample per second: 14207\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.93% ) in 30.09 sec. Users per second: 1595\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.68 sec. Users per second: 1631\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1741694, PRECISION: 0.0555166, PRECISION_RECALL_MIN_DEN: 0.0731172, RECALL: 0.0516779, MAP: 0.0336380, MRR: 0.1408257, NDCG: 0.0473994, F1: 0.0535285, HIT_RATE: 0.5551662, ARHR: 0.1889704, NOVELTY: 0.0112153, AVERAGE_POPULARITY: 0.3724127, DIVERSITY_MEAN_INTER_LIST: 0.9304538, DIVERSITY_HERFINDAHL: 0.9930440, COVERAGE_ITEM: 0.9993446, COVERAGE_ITEM_CORRECT: 0.0770527, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3031569, DIVERSITY_GINI: 0.3177041, SHANNON_ENTROPY: 10.2298142, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 155 of 1500. Elapsed time 32.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 5.60E-01. Sample per second: 13568\n",
      "MF_BPR: Epoch 156 of 1500. Elapsed time 32.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 5.85E-01. Sample per second: 15361\n",
      "MF_BPR: Epoch 157 of 1500. Elapsed time 32.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 6.02E-01. Sample per second: 14320\n",
      "MF_BPR: Epoch 158 of 1500. Elapsed time 32.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.18 seconds. BPR loss 6.11E-01. Sample per second: 13478\n",
      "MF_BPR: Epoch 159 of 1500. Elapsed time 32.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 6.17E-01. Sample per second: 15575\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.06 sec. Users per second: 1663\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.95 sec. Users per second: 1660\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1791192, PRECISION: 0.0573993, PRECISION_RECALL_MIN_DEN: 0.0752966, RECALL: 0.0529035, MAP: 0.0346809, MRR: 0.1448776, NDCG: 0.0487012, F1: 0.0550598, HIT_RATE: 0.5739930, ARHR: 0.1949504, NOVELTY: 0.0110776, AVERAGE_POPULARITY: 0.3863643, DIVERSITY_MEAN_INTER_LIST: 0.9251053, DIVERSITY_HERFINDAHL: 0.9925092, COVERAGE_ITEM: 0.9991574, COVERAGE_ITEM_CORRECT: 0.0758356, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3116718, DIVERSITY_GINI: 0.3029052, SHANNON_ENTROPY: 10.0758996, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 160 of 1500. Elapsed time 33.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 6.56E-01. Sample per second: 14346\n",
      "MF_BPR: Epoch 161 of 1500. Elapsed time 33.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.26 seconds. BPR loss 6.67E-01. Sample per second: 13291\n",
      "MF_BPR: Epoch 162 of 1500. Elapsed time 33.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 6.78E-01. Sample per second: 15611\n",
      "MF_BPR: Epoch 163 of 1500. Elapsed time 33.55 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 7.08E-01. Sample per second: 15858\n",
      "MF_BPR: Epoch 164 of 1500. Elapsed time 33.62 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 7.26E-01. Sample per second: 15818\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.00 sec. Users per second: 1966\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.19 sec. Users per second: 1979\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1829388, PRECISION: 0.0593078, PRECISION_RECALL_MIN_DEN: 0.0775251, RECALL: 0.0541471, MAP: 0.0356910, MRR: 0.1485492, NDCG: 0.0499689, F1: 0.0566101, HIT_RATE: 0.5930782, ARHR: 0.2005751, NOVELTY: 0.0109535, AVERAGE_POPULARITY: 0.3991597, DIVERSITY_MEAN_INTER_LIST: 0.9199365, DIVERSITY_HERFINDAHL: 0.9919923, COVERAGE_ITEM: 0.9992510, COVERAGE_ITEM_CORRECT: 0.0749930, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3197144, DIVERSITY_GINI: 0.2893210, SHANNON_ENTROPY: 9.9331111, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 165 of 1500. Elapsed time 34.27 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 7.30E-01. Sample per second: 15235\n",
      "MF_BPR: Epoch 166 of 1500. Elapsed time 34.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 7.66E-01. Sample per second: 15821\n",
      "MF_BPR: Epoch 167 of 1500. Elapsed time 34.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 7.82E-01. Sample per second: 16482\n",
      "MF_BPR: Epoch 168 of 1500. Elapsed time 34.46 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.11 seconds. BPR loss 8.17E-01. Sample per second: 17018\n",
      "MF_BPR: Epoch 169 of 1500. Elapsed time 34.53 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 3.94 seconds. BPR loss 8.24E-01. Sample per second: 17729\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 59000 ( 84.73% ) in 30.13 sec. Users per second: 1958\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 35.35 sec. Users per second: 1970\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1876338, PRECISION: 0.0609363, PRECISION_RECALL_MIN_DEN: 0.0794315, RECALL: 0.0551642, MAP: 0.0365883, MRR: 0.1520361, NDCG: 0.0510824, F1: 0.0579068, HIT_RATE: 0.6093631, ARHR: 0.2057191, NOVELTY: 0.0108413, AVERAGE_POPULARITY: 0.4109046, DIVERSITY_MEAN_INTER_LIST: 0.9151345, DIVERSITY_HERFINDAHL: 0.9915121, COVERAGE_ITEM: 0.9990638, COVERAGE_ITEM_CORRECT: 0.0734950, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3275709, DIVERSITY_GINI: 0.2771285, SHANNON_ENTROPY: 9.8021525, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 170 of 1500. Elapsed time 35.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.25 seconds. BPR loss 8.56E-01. Sample per second: 16434\n",
      "MF_BPR: Epoch 171 of 1500. Elapsed time 35.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.15 seconds. BPR loss 8.92E-01. Sample per second: 16834\n",
      "MF_BPR: Epoch 172 of 1500. Elapsed time 35.31 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.10 seconds. BPR loss 8.88E-01. Sample per second: 17035\n",
      "MF_BPR: Epoch 173 of 1500. Elapsed time 35.38 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.13 seconds. BPR loss 9.25E-01. Sample per second: 16900\n",
      "MF_BPR: Epoch 174 of 1500. Elapsed time 35.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.06 seconds. BPR loss 9.52E-01. Sample per second: 17222\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 58000 ( 83.29% ) in 30.47 sec. Users per second: 1903\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.11 sec. Users per second: 1877\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1912435, PRECISION: 0.0624801, PRECISION_RECALL_MIN_DEN: 0.0812500, RECALL: 0.0561366, MAP: 0.0374263, MRR: 0.1552953, NDCG: 0.0521118, F1: 0.0591387, HIT_RATE: 0.6248007, ARHR: 0.2105625, NOVELTY: 0.0107373, AVERAGE_POPULARITY: 0.4217846, DIVERSITY_MEAN_INTER_LIST: 0.9106078, DIVERSITY_HERFINDAHL: 0.9910595, COVERAGE_ITEM: 0.9990638, COVERAGE_ITEM_CORRECT: 0.0714353, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3342397, DIVERSITY_GINI: 0.2659105, SHANNON_ENTROPY: 9.6796482, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 175 of 1500. Elapsed time 36.13 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 9.65E-01. Sample per second: 15614\n",
      "MF_BPR: Epoch 176 of 1500. Elapsed time 36.20 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 9.90E-01. Sample per second: 14420\n",
      "MF_BPR: Epoch 177 of 1500. Elapsed time 36.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.02E+00. Sample per second: 13879\n",
      "MF_BPR: Epoch 178 of 1500. Elapsed time 36.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 1.03E+00. Sample per second: 16541\n",
      "MF_BPR: Epoch 179 of 1500. Elapsed time 36.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.52 seconds. BPR loss 1.04E+00. Sample per second: 15470\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.22 sec. Users per second: 1820\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.95 sec. Users per second: 1835\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1949204, PRECISION: 0.0638601, PRECISION_RECALL_MIN_DEN: 0.0828702, RECALL: 0.0569677, MAP: 0.0381974, MRR: 0.1582942, NDCG: 0.0530678, F1: 0.0602173, HIT_RATE: 0.6386013, ARHR: 0.2150382, NOVELTY: 0.0106428, AVERAGE_POPULARITY: 0.4320963, DIVERSITY_MEAN_INTER_LIST: 0.9063285, DIVERSITY_HERFINDAHL: 0.9906316, COVERAGE_ITEM: 0.9990638, COVERAGE_ITEM_CORRECT: 0.0688138, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3401929, DIVERSITY_GINI: 0.2555314, SHANNON_ENTROPY: 9.5670360, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 180 of 1500. Elapsed time 37.12 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.08E+00. Sample per second: 14731\n",
      "MF_BPR: Epoch 181 of 1500. Elapsed time 37.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 1.09E+00. Sample per second: 14386\n",
      "MF_BPR: Epoch 182 of 1500. Elapsed time 37.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 1.14E+00. Sample per second: 13988\n",
      "MF_BPR: Epoch 183 of 1500. Elapsed time 37.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 1.17E+00. Sample per second: 13734\n",
      "MF_BPR: Epoch 184 of 1500. Elapsed time 37.39 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 1.21E+00. Sample per second: 16586\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.28 sec. Users per second: 1850\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.65 sec. Users per second: 1849\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.1978235, PRECISION: 0.0653350, PRECISION_RECALL_MIN_DEN: 0.0845758, RECALL: 0.0578653, MAP: 0.0389298, MRR: 0.1611354, NDCG: 0.0540675, F1: 0.0613737, HIT_RATE: 0.6533496, ARHR: 0.2193307, NOVELTY: 0.0105574, AVERAGE_POPULARITY: 0.4415551, DIVERSITY_MEAN_INTER_LIST: 0.9023193, DIVERSITY_HERFINDAHL: 0.9902306, COVERAGE_ITEM: 0.9990638, COVERAGE_ITEM_CORRECT: 0.0681584, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3465611, DIVERSITY_GINI: 0.2464833, SHANNON_ENTROPY: 9.4641144, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 185 of 1500. Elapsed time 38.09 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 1.22E+00. Sample per second: 13765\n",
      "MF_BPR: Epoch 186 of 1500. Elapsed time 38.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.23 seconds. BPR loss 1.25E+00. Sample per second: 16530\n",
      "MF_BPR: Epoch 187 of 1500. Elapsed time 38.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.29E+00. Sample per second: 15524\n",
      "MF_BPR: Epoch 188 of 1500. Elapsed time 38.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 1.30E+00. Sample per second: 14803\n",
      "MF_BPR: Epoch 189 of 1500. Elapsed time 38.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.32E+00. Sample per second: 14221\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.07 sec. Users per second: 1696\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.35 sec. Users per second: 1726\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2006936, PRECISION: 0.0664881, PRECISION_RECALL_MIN_DEN: 0.0860019, RECALL: 0.0586362, MAP: 0.0395301, MRR: 0.1633792, NDCG: 0.0548818, F1: 0.0623158, HIT_RATE: 0.6648812, ARHR: 0.2228171, NOVELTY: 0.0104793, AVERAGE_POPULARITY: 0.4502069, DIVERSITY_MEAN_INTER_LIST: 0.8984605, DIVERSITY_HERFINDAHL: 0.9898448, COVERAGE_ITEM: 0.9988765, COVERAGE_ITEM_CORRECT: 0.0672222, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3514268, DIVERSITY_GINI: 0.2381258, SHANNON_ENTROPY: 9.3697018, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 190 of 1500. Elapsed time 39.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.37E+00. Sample per second: 15517\n",
      "MF_BPR: Epoch 191 of 1500. Elapsed time 39.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 1.38E+00. Sample per second: 15090\n",
      "MF_BPR: Epoch 192 of 1500. Elapsed time 39.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 1.43E+00. Sample per second: 14595\n",
      "MF_BPR: Epoch 193 of 1500. Elapsed time 39.32 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 1.43E+00. Sample per second: 14112\n",
      "MF_BPR: Epoch 194 of 1500. Elapsed time 39.39 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 1.49E+00. Sample per second: 13733\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.12 sec. Users per second: 1826\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.01 sec. Users per second: 1832\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2036661, PRECISION: 0.0675795, PRECISION_RECALL_MIN_DEN: 0.0872244, RECALL: 0.0592338, MAP: 0.0401644, MRR: 0.1659330, NDCG: 0.0556357, F1: 0.0631321, HIT_RATE: 0.6757952, ARHR: 0.2266012, NOVELTY: 0.0104099, AVERAGE_POPULARITY: 0.4581470, DIVERSITY_MEAN_INTER_LIST: 0.8949729, DIVERSITY_HERFINDAHL: 0.9894960, COVERAGE_ITEM: 0.9988765, COVERAGE_ITEM_CORRECT: 0.0658178, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3563926, DIVERSITY_GINI: 0.2307025, SHANNON_ENTROPY: 9.2846630, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 195 of 1500. Elapsed time 40.10 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.49E+00. Sample per second: 15674\n",
      "MF_BPR: Epoch 196 of 1500. Elapsed time 40.17 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 1.52E+00. Sample per second: 14977\n",
      "MF_BPR: Epoch 197 of 1500. Elapsed time 40.24 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 1.55E+00. Sample per second: 14577\n",
      "MF_BPR: Epoch 198 of 1500. Elapsed time 40.31 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 1.59E+00. Sample per second: 14310\n",
      "MF_BPR: Epoch 199 of 1500. Elapsed time 40.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.62E+00. Sample per second: 13891\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.26 sec. Users per second: 1784\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.00 sec. Users per second: 1786\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2062623, PRECISION: 0.0685991, PRECISION_RECALL_MIN_DEN: 0.0883786, RECALL: 0.0598261, MAP: 0.0407128, MRR: 0.1681585, NDCG: 0.0563395, F1: 0.0639130, HIT_RATE: 0.6859912, ARHR: 0.2299480, NOVELTY: 0.0103470, AVERAGE_POPULARITY: 0.4655546, DIVERSITY_MEAN_INTER_LIST: 0.8916420, DIVERSITY_HERFINDAHL: 0.9891629, COVERAGE_ITEM: 0.9986893, COVERAGE_ITEM_CORRECT: 0.0645071, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3612439, DIVERSITY_GINI: 0.2239769, SHANNON_ENTROPY: 9.2078442, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 200 of 1500. Elapsed time 41.09 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 1.64E+00. Sample per second: 16193\n",
      "MF_BPR: Epoch 201 of 1500. Elapsed time 41.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.70E+00. Sample per second: 15666\n",
      "MF_BPR: Epoch 202 of 1500. Elapsed time 41.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 1.72E+00. Sample per second: 15013\n",
      "MF_BPR: Epoch 203 of 1500. Elapsed time 41.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 1.73E+00. Sample per second: 14488\n",
      "MF_BPR: Epoch 204 of 1500. Elapsed time 41.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 1.79E+00. Sample per second: 13910\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.36 sec. Users per second: 1713\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.58 sec. Users per second: 1716\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2084643, PRECISION: 0.0695570, PRECISION_RECALL_MIN_DEN: 0.0895420, RECALL: 0.0604474, MAP: 0.0412254, MRR: 0.1701281, NDCG: 0.0570825, F1: 0.0646830, HIT_RATE: 0.6955698, ARHR: 0.2329897, NOVELTY: 0.0102897, AVERAGE_POPULARITY: 0.4723608, DIVERSITY_MEAN_INTER_LIST: 0.8886412, DIVERSITY_HERFINDAHL: 0.9888628, COVERAGE_ITEM: 0.9986893, COVERAGE_ITEM_CORRECT: 0.0631963, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3653510, DIVERSITY_GINI: 0.2181195, SHANNON_ENTROPY: 9.1381286, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 205 of 1500. Elapsed time 42.12 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.79E+00. Sample per second: 14237\n",
      "MF_BPR: Epoch 206 of 1500. Elapsed time 42.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.82E+00. Sample per second: 13689\n",
      "MF_BPR: Epoch 207 of 1500. Elapsed time 42.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 1.84E+00. Sample per second: 15910\n",
      "MF_BPR: Epoch 208 of 1500. Elapsed time 42.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 1.86E+00. Sample per second: 14916\n",
      "MF_BPR: Epoch 209 of 1500. Elapsed time 42.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.91E+00. Sample per second: 14254\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.29 sec. Users per second: 1684\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.89 sec. Users per second: 1703\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2106434, PRECISION: 0.0705148, PRECISION_RECALL_MIN_DEN: 0.0905594, RECALL: 0.0608994, MAP: 0.0417110, MRR: 0.1721554, NDCG: 0.0577894, F1: 0.0653554, HIT_RATE: 0.7051483, ARHR: 0.2360196, NOVELTY: 0.0102370, AVERAGE_POPULARITY: 0.4787833, DIVERSITY_MEAN_INTER_LIST: 0.8858641, DIVERSITY_HERFINDAHL: 0.9885851, COVERAGE_ITEM: 0.9987829, COVERAGE_ITEM_CORRECT: 0.0628218, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3697730, DIVERSITY_GINI: 0.2125329, SHANNON_ENTROPY: 9.0739729, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 210 of 1500. Elapsed time 43.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.11 seconds. BPR loss 1.98E+00. Sample per second: 13682\n",
      "MF_BPR: Epoch 211 of 1500. Elapsed time 43.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 2.00E+00. Sample per second: 16129\n",
      "MF_BPR: Epoch 212 of 1500. Elapsed time 43.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 2.00E+00. Sample per second: 15189\n",
      "MF_BPR: Epoch 213 of 1500. Elapsed time 43.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 2.09E+00. Sample per second: 14360\n",
      "MF_BPR: Epoch 214 of 1500. Elapsed time 43.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 2.10E+00. Sample per second: 13712\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.36 sec. Users per second: 1811\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.58 sec. Users per second: 1805\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2132889, PRECISION: 0.0712946, PRECISION_RECALL_MIN_DEN: 0.0913700, RECALL: 0.0612540, MAP: 0.0421386, MRR: 0.1741164, NDCG: 0.0583768, F1: 0.0658940, HIT_RATE: 0.7129461, ARHR: 0.2388128, NOVELTY: 0.0101899, AVERAGE_POPULARITY: 0.4847198, DIVERSITY_MEAN_INTER_LIST: 0.8832418, DIVERSITY_HERFINDAHL: 0.9883229, COVERAGE_ITEM: 0.9985020, COVERAGE_ITEM_CORRECT: 0.0618856, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3735654, DIVERSITY_GINI: 0.2075979, SHANNON_ENTROPY: 9.0160697, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 215 of 1500. Elapsed time 44.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 2.15E+00. Sample per second: 13926\n",
      "MF_BPR: Epoch 216 of 1500. Elapsed time 44.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.26 seconds. BPR loss 2.18E+00. Sample per second: 16399\n",
      "MF_BPR: Epoch 217 of 1500. Elapsed time 44.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 2.18E+00. Sample per second: 15919\n",
      "MF_BPR: Epoch 218 of 1500. Elapsed time 44.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 2.22E+00. Sample per second: 15301\n",
      "MF_BPR: Epoch 219 of 1500. Elapsed time 44.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 2.24E+00. Sample per second: 14818\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.29 sec. Users per second: 1750\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.23 sec. Users per second: 1775\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2149220, PRECISION: 0.0720313, PRECISION_RECALL_MIN_DEN: 0.0921908, RECALL: 0.0616154, MAP: 0.0425390, MRR: 0.1756942, NDCG: 0.0589283, F1: 0.0664175, HIT_RATE: 0.7203131, ARHR: 0.2412010, NOVELTY: 0.0101479, AVERAGE_POPULARITY: 0.4900411, DIVERSITY_MEAN_INTER_LIST: 0.8807755, DIVERSITY_HERFINDAHL: 0.9880763, COVERAGE_ITEM: 0.9984084, COVERAGE_ITEM_CORRECT: 0.0616983, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3767280, DIVERSITY_GINI: 0.2031047, SHANNON_ENTROPY: 8.9639242, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 220 of 1500. Elapsed time 45.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.20 seconds. BPR loss 2.28E+00. Sample per second: 16656\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 221 of 1500. Elapsed time 45.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 2.30E+00. Sample per second: 16042\n",
      "MF_BPR: Epoch 222 of 1500. Elapsed time 45.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 2.33E+00. Sample per second: 15579\n",
      "MF_BPR: Epoch 223 of 1500. Elapsed time 45.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 2.39E+00. Sample per second: 15217\n",
      "MF_BPR: Epoch 224 of 1500. Elapsed time 45.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 2.38E+00. Sample per second: 14703\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.43 sec. Users per second: 1808\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.18 sec. Users per second: 1824\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2170591, PRECISION: 0.0726373, PRECISION_RECALL_MIN_DEN: 0.0928926, RECALL: 0.0619453, MAP: 0.0429206, MRR: 0.1773610, NDCG: 0.0594240, F1: 0.0668666, HIT_RATE: 0.7263732, ARHR: 0.2435967, NOVELTY: 0.0101096, AVERAGE_POPULARITY: 0.4947510, DIVERSITY_MEAN_INTER_LIST: 0.8787557, DIVERSITY_HERFINDAHL: 0.9878743, COVERAGE_ITEM: 0.9981275, COVERAGE_ITEM_CORRECT: 0.0605749, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3792896, DIVERSITY_GINI: 0.1989196, SHANNON_ENTROPY: 8.9182390, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 225 of 1500. Elapsed time 46.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.13 seconds. BPR loss 2.45E+00. Sample per second: 16904\n",
      "MF_BPR: Epoch 226 of 1500. Elapsed time 46.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 2.47E+00. Sample per second: 16230\n",
      "MF_BPR: Epoch 227 of 1500. Elapsed time 46.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 2.52E+00. Sample per second: 15675\n",
      "MF_BPR: Epoch 228 of 1500. Elapsed time 46.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 2.51E+00. Sample per second: 15232\n",
      "MF_BPR: Epoch 229 of 1500. Elapsed time 46.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 2.56E+00. Sample per second: 14819\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.29 sec. Users per second: 1783\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.83 sec. Users per second: 1793\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2185634, PRECISION: 0.0732821, PRECISION_RECALL_MIN_DEN: 0.0936479, RECALL: 0.0623165, MAP: 0.0432466, MRR: 0.1786330, NDCG: 0.0599224, F1: 0.0673559, HIT_RATE: 0.7328211, ARHR: 0.2455345, NOVELTY: 0.0100750, AVERAGE_POPULARITY: 0.4991751, DIVERSITY_MEAN_INTER_LIST: 0.8768703, DIVERSITY_HERFINDAHL: 0.9876858, COVERAGE_ITEM: 0.9978466, COVERAGE_ITEM_CORRECT: 0.0595450, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3820802, DIVERSITY_GINI: 0.1952408, SHANNON_ENTROPY: 8.8760685, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 230 of 1500. Elapsed time 47.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 2.62E+00. Sample per second: 14445\n",
      "MF_BPR: Epoch 231 of 1500. Elapsed time 47.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 2.67E+00. Sample per second: 14012\n",
      "MF_BPR: Epoch 232 of 1500. Elapsed time 47.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.17 seconds. BPR loss 2.71E+00. Sample per second: 13523\n",
      "MF_BPR: Epoch 233 of 1500. Elapsed time 47.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 2.76E+00. Sample per second: 16262\n",
      "MF_BPR: Epoch 234 of 1500. Elapsed time 47.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 2.74E+00. Sample per second: 15749\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.37 sec. Users per second: 1745\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.23 sec. Users per second: 1731\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2200052, PRECISION: 0.0738881, PRECISION_RECALL_MIN_DEN: 0.0943594, RECALL: 0.0626804, MAP: 0.0434989, MRR: 0.1796774, NDCG: 0.0603565, F1: 0.0678244, HIT_RATE: 0.7388813, ARHR: 0.2471626, NOVELTY: 0.0100437, AVERAGE_POPULARITY: 0.5030722, DIVERSITY_MEAN_INTER_LIST: 0.8751571, DIVERSITY_HERFINDAHL: 0.9875145, COVERAGE_ITEM: 0.9976594, COVERAGE_ITEM_CORRECT: 0.0586087, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3850568, DIVERSITY_GINI: 0.1919595, SHANNON_ENTROPY: 8.8388051, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 235 of 1500. Elapsed time 48.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 2.82E+00. Sample per second: 13777\n",
      "MF_BPR: Epoch 236 of 1500. Elapsed time 48.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 2.86E+00. Sample per second: 16038\n",
      "MF_BPR: Epoch 237 of 1500. Elapsed time 48.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 2.85E+00. Sample per second: 14809\n",
      "MF_BPR: Epoch 238 of 1500. Elapsed time 48.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 2.89E+00. Sample per second: 13901\n",
      "MF_BPR: Epoch 239 of 1500. Elapsed time 48.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 2.90E+00. Sample per second: 16243\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.93% ) in 30.18 sec. Users per second: 1590\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.95 sec. Users per second: 1621\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2215088, PRECISION: 0.0743649, PRECISION_RECALL_MIN_DEN: 0.0948630, RECALL: 0.0628682, MAP: 0.0437539, MRR: 0.1810132, NDCG: 0.0607403, F1: 0.0681350, HIT_RATE: 0.7436490, ARHR: 0.2489526, NOVELTY: 0.0100159, AVERAGE_POPULARITY: 0.5066710, DIVERSITY_MEAN_INTER_LIST: 0.8736593, DIVERSITY_HERFINDAHL: 0.9873647, COVERAGE_ITEM: 0.9977530, COVERAGE_ITEM_CORRECT: 0.0587024, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3873751, DIVERSITY_GINI: 0.1890199, SHANNON_ENTROPY: 8.8058512, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 240 of 1500. Elapsed time 49.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 2.97E+00. Sample per second: 14960\n",
      "MF_BPR: Epoch 241 of 1500. Elapsed time 49.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 3.02E+00. Sample per second: 14246\n",
      "MF_BPR: Epoch 242 of 1500. Elapsed time 49.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 3.07E+00. Sample per second: 13654\n",
      "MF_BPR: Epoch 243 of 1500. Elapsed time 49.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 3.06E+00. Sample per second: 16104\n",
      "MF_BPR: Epoch 244 of 1500. Elapsed time 49.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 3.10E+00. Sample per second: 15300\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.11 sec. Users per second: 1661\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.37 sec. Users per second: 1683\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2225966, PRECISION: 0.0748919, PRECISION_RECALL_MIN_DEN: 0.0954321, RECALL: 0.0631089, MAP: 0.0440234, MRR: 0.1821135, NDCG: 0.0611531, F1: 0.0684974, HIT_RATE: 0.7489194, ARHR: 0.2506377, NOVELTY: 0.0099898, AVERAGE_POPULARITY: 0.5102371, DIVERSITY_MEAN_INTER_LIST: 0.8719842, DIVERSITY_HERFINDAHL: 0.9871972, COVERAGE_ITEM: 0.9975658, COVERAGE_ITEM_CORRECT: 0.0577661, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3896362, DIVERSITY_GINI: 0.1862901, SHANNON_ENTROPY: 8.7736193, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 245 of 1500. Elapsed time 50.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 3.14E+00. Sample per second: 16271\n",
      "MF_BPR: Epoch 246 of 1500. Elapsed time 50.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 3.25E+00. Sample per second: 15393\n",
      "MF_BPR: Epoch 247 of 1500. Elapsed time 50.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 3.26E+00. Sample per second: 14617\n",
      "MF_BPR: Epoch 248 of 1500. Elapsed time 50.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 3.38E+00. Sample per second: 13883\n",
      "MF_BPR: Epoch 249 of 1500. Elapsed time 50.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 3.31E+00. Sample per second: 16330\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.31 sec. Users per second: 1649\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.53 sec. Users per second: 1677\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2236002, PRECISION: 0.0752883, PRECISION_RECALL_MIN_DEN: 0.0959039, RECALL: 0.0633341, MAP: 0.0442057, MRR: 0.1828359, NDCG: 0.0614840, F1: 0.0687958, HIT_RATE: 0.7528829, ARHR: 0.2517837, NOVELTY: 0.0099676, AVERAGE_POPULARITY: 0.5132677, DIVERSITY_MEAN_INTER_LIST: 0.8704930, DIVERSITY_HERFINDAHL: 0.9870480, COVERAGE_ITEM: 0.9974721, COVERAGE_ITEM_CORRECT: 0.0576725, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3916826, DIVERSITY_GINI: 0.1837465, SHANNON_ENTROPY: 8.7454531, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 250 of 1500. Elapsed time 51.32 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 3.35E+00. Sample per second: 13658\n",
      "MF_BPR: Epoch 251 of 1500. Elapsed time 51.39 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 3.41E+00. Sample per second: 16178\n",
      "MF_BPR: Epoch 252 of 1500. Elapsed time 51.46 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 3.47E+00. Sample per second: 15523\n",
      "MF_BPR: Epoch 253 of 1500. Elapsed time 51.53 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 3.49E+00. Sample per second: 14830\n",
      "MF_BPR: Epoch 254 of 1500. Elapsed time 51.60 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 3.50E+00. Sample per second: 14249\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.18 sec. Users per second: 1723\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.43 sec. Users per second: 1722\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2245658, PRECISION: 0.0757306, PRECISION_RECALL_MIN_DEN: 0.0964097, RECALL: 0.0635475, MAP: 0.0444039, MRR: 0.1838034, NDCG: 0.0618577, F1: 0.0691062, HIT_RATE: 0.7573060, ARHR: 0.2531514, NOVELTY: 0.0099467, AVERAGE_POPULARITY: 0.5159910, DIVERSITY_MEAN_INTER_LIST: 0.8694132, DIVERSITY_HERFINDAHL: 0.9869401, COVERAGE_ITEM: 0.9971913, COVERAGE_ITEM_CORRECT: 0.0570171, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3936432, DIVERSITY_GINI: 0.1815820, SHANNON_ENTROPY: 8.7217927, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 255 of 1500. Elapsed time 52.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 3.52E+00. Sample per second: 14998\n",
      "MF_BPR: Epoch 256 of 1500. Elapsed time 52.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 3.62E+00. Sample per second: 14312\n",
      "MF_BPR: Epoch 257 of 1500. Elapsed time 52.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 3.60E+00. Sample per second: 13870\n",
      "MF_BPR: Epoch 258 of 1500. Elapsed time 52.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.23 seconds. BPR loss 3.69E+00. Sample per second: 16526\n",
      "MF_BPR: Epoch 259 of 1500. Elapsed time 52.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 3.70E+00. Sample per second: 15794\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.46 sec. Users per second: 1707\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.53 sec. Users per second: 1718\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2253966, PRECISION: 0.0762102, PRECISION_RECALL_MIN_DEN: 0.0969213, RECALL: 0.0637519, MAP: 0.0446290, MRR: 0.1847794, NDCG: 0.0622344, F1: 0.0694266, HIT_RATE: 0.7621024, ARHR: 0.2545438, NOVELTY: 0.0099266, AVERAGE_POPULARITY: 0.5186139, DIVERSITY_MEAN_INTER_LIST: 0.8682353, DIVERSITY_HERFINDAHL: 0.9868223, COVERAGE_ITEM: 0.9974721, COVERAGE_ITEM_CORRECT: 0.0562681, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3959043, DIVERSITY_GINI: 0.1793407, SHANNON_ENTROPY: 8.6992020, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 260 of 1500. Elapsed time 53.38 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.25 seconds. BPR loss 3.72E+00. Sample per second: 16460\n",
      "MF_BPR: Epoch 261 of 1500. Elapsed time 53.45 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 3.79E+00. Sample per second: 15797\n",
      "MF_BPR: Epoch 262 of 1500. Elapsed time 53.52 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 3.76E+00. Sample per second: 15052\n",
      "MF_BPR: Epoch 263 of 1500. Elapsed time 53.59 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 3.87E+00. Sample per second: 14422\n",
      "MF_BPR: Epoch 264 of 1500. Elapsed time 53.66 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 3.86E+00. Sample per second: 13862\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.10 sec. Users per second: 1694\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.13 sec. Users per second: 1735\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2262947, PRECISION: 0.0765477, PRECISION_RECALL_MIN_DEN: 0.0972303, RECALL: 0.0638084, MAP: 0.0447707, MRR: 0.1853851, NDCG: 0.0624816, F1: 0.0695999, HIT_RATE: 0.7654771, ARHR: 0.2554593, NOVELTY: 0.0099090, AVERAGE_POPULARITY: 0.5210268, DIVERSITY_MEAN_INTER_LIST: 0.8670412, DIVERSITY_HERFINDAHL: 0.9867029, COVERAGE_ITEM: 0.9975658, COVERAGE_ITEM_CORRECT: 0.0560809, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3974498, DIVERSITY_GINI: 0.1774362, SHANNON_ENTROPY: 8.6790848, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 265 of 1500. Elapsed time 54.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 3.88E+00. Sample per second: 15856\n",
      "MF_BPR: Epoch 266 of 1500. Elapsed time 54.47 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 3.92E+00. Sample per second: 15524\n",
      "MF_BPR: Epoch 267 of 1500. Elapsed time 54.53 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 4.01E+00. Sample per second: 15164\n",
      "MF_BPR: Epoch 268 of 1500. Elapsed time 54.60 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 4.10E+00. Sample per second: 14440\n",
      "MF_BPR: Epoch 269 of 1500. Elapsed time 54.67 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 4.11E+00. Sample per second: 13972\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.15 sec. Users per second: 1758\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.41 sec. Users per second: 1767\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2270681, PRECISION: 0.0768234, PRECISION_RECALL_MIN_DEN: 0.0974873, RECALL: 0.0638890, MAP: 0.0449587, MRR: 0.1864226, NDCG: 0.0628082, F1: 0.0697617, HIT_RATE: 0.7682344, ARHR: 0.2568218, NOVELTY: 0.0098938, AVERAGE_POPULARITY: 0.5230806, DIVERSITY_MEAN_INTER_LIST: 0.8662496, DIVERSITY_HERFINDAHL: 0.9866237, COVERAGE_ITEM: 0.9974721, COVERAGE_ITEM_CORRECT: 0.0557064, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3985231, DIVERSITY_GINI: 0.1757892, SHANNON_ENTROPY: 8.6620551, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 270 of 1500. Elapsed time 55.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 4.13E+00. Sample per second: 14884\n",
      "MF_BPR: Epoch 271 of 1500. Elapsed time 55.47 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 4.15E+00. Sample per second: 14291\n",
      "MF_BPR: Epoch 272 of 1500. Elapsed time 55.54 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 4.17E+00. Sample per second: 13653\n",
      "MF_BPR: Epoch 273 of 1500. Elapsed time 55.61 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 4.20E+00. Sample per second: 16199\n",
      "MF_BPR: Epoch 274 of 1500. Elapsed time 55.68 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 4.23E+00. Sample per second: 15425\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.30 sec. Users per second: 1716\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.50 sec. Users per second: 1719\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2277193, PRECISION: 0.0772040, PRECISION_RECALL_MIN_DEN: 0.0978828, RECALL: 0.0640560, MAP: 0.0451055, MRR: 0.1868980, NDCG: 0.0630937, F1: 0.0700181, HIT_RATE: 0.7720399, ARHR: 0.2576693, NOVELTY: 0.0098788, AVERAGE_POPULARITY: 0.5249355, DIVERSITY_MEAN_INTER_LIST: 0.8655662, DIVERSITY_HERFINDAHL: 0.9865554, COVERAGE_ITEM: 0.9972849, COVERAGE_ITEM_CORRECT: 0.0554255, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.3999113, DIVERSITY_GINI: 0.1741191, SHANNON_ENTROPY: 8.6463730, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 275 of 1500. Elapsed time 56.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 4.24E+00. Sample per second: 15804\n",
      "MF_BPR: Epoch 276 of 1500. Elapsed time 56.50 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 4.30E+00. Sample per second: 14828\n",
      "MF_BPR: Epoch 277 of 1500. Elapsed time 56.57 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 4.33E+00. Sample per second: 13908\n",
      "MF_BPR: Epoch 278 of 1500. Elapsed time 56.64 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 4.38E+00. Sample per second: 16567\n",
      "MF_BPR: Epoch 279 of 1500. Elapsed time 56.71 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 4.46E+00. Sample per second: 15861\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.40 sec. Users per second: 1677\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.91 sec. Users per second: 1661\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2281576, PRECISION: 0.0774639, PRECISION_RECALL_MIN_DEN: 0.0981036, RECALL: 0.0640989, MAP: 0.0452327, MRR: 0.1874454, NDCG: 0.0633070, F1: 0.0701505, HIT_RATE: 0.7746392, ARHR: 0.2584587, NOVELTY: 0.0098666, AVERAGE_POPULARITY: 0.5264821, DIVERSITY_MEAN_INTER_LIST: 0.8648971, DIVERSITY_HERFINDAHL: 0.9864885, COVERAGE_ITEM: 0.9969104, COVERAGE_ITEM_CORRECT: 0.0549574, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4009989, DIVERSITY_GINI: 0.1725984, SHANNON_ENTROPY: 8.6333651, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 280 of 1500. Elapsed time 57.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 4.52E+00. Sample per second: 14483\n",
      "MF_BPR: Epoch 281 of 1500. Elapsed time 57.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.18 seconds. BPR loss 4.53E+00. Sample per second: 13490\n",
      "MF_BPR: Epoch 282 of 1500. Elapsed time 57.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.43 seconds. BPR loss 4.52E+00. Sample per second: 15757\n",
      "MF_BPR: Epoch 283 of 1500. Elapsed time 57.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 4.60E+00. Sample per second: 14716\n",
      "MF_BPR: Epoch 284 of 1500. Elapsed time 57.77 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 4.63E+00. Sample per second: 13781\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.51 sec. Users per second: 1639\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.08 sec. Users per second: 1655\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2287953, PRECISION: 0.0777109, PRECISION_RECALL_MIN_DEN: 0.0983800, RECALL: 0.0642312, MAP: 0.0453365, MRR: 0.1878868, NDCG: 0.0635598, F1: 0.0703310, HIT_RATE: 0.7771092, ARHR: 0.2591583, NOVELTY: 0.0098545, AVERAGE_POPULARITY: 0.5279773, DIVERSITY_MEAN_INTER_LIST: 0.8642935, DIVERSITY_HERFINDAHL: 0.9864281, COVERAGE_ITEM: 0.9968168, COVERAGE_ITEM_CORRECT: 0.0549574, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4023298, DIVERSITY_GINI: 0.1710852, SHANNON_ENTROPY: 8.6209409, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 285 of 1500. Elapsed time 58.55 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 4.60E+00. Sample per second: 14978\n",
      "MF_BPR: Epoch 286 of 1500. Elapsed time 58.62 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 4.72E+00. Sample per second: 13719\n",
      "MF_BPR: Epoch 287 of 1500. Elapsed time 58.69 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 4.69E+00. Sample per second: 16165\n",
      "MF_BPR: Epoch 288 of 1500. Elapsed time 58.76 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 4.81E+00. Sample per second: 14976\n",
      "MF_BPR: Epoch 289 of 1500. Elapsed time 58.84 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 4.76E+00. Sample per second: 14007\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.12 sec. Users per second: 1826\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.91 sec. Users per second: 1837\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2290811, PRECISION: 0.0780225, PRECISION_RECALL_MIN_DEN: 0.0986896, RECALL: 0.0643432, MAP: 0.0454749, MRR: 0.1884163, NDCG: 0.0638271, F1: 0.0705257, HIT_RATE: 0.7802255, ARHR: 0.2600604, NOVELTY: 0.0098435, AVERAGE_POPULARITY: 0.5293819, DIVERSITY_MEAN_INTER_LIST: 0.8637168, DIVERSITY_HERFINDAHL: 0.9863704, COVERAGE_ITEM: 0.9970040, COVERAGE_ITEM_CORRECT: 0.0546765, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4035032, DIVERSITY_GINI: 0.1698632, SHANNON_ENTROPY: 8.6099299, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 290 of 1500. Elapsed time 59.54 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.13 seconds. BPR loss 4.76E+00. Sample per second: 13630\n",
      "MF_BPR: Epoch 291 of 1500. Elapsed time 59.61 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 4.87E+00. Sample per second: 16173\n",
      "MF_BPR: Epoch 292 of 1500. Elapsed time 59.68 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 4.94E+00. Sample per second: 15664\n",
      "MF_BPR: Epoch 293 of 1500. Elapsed time 59.75 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 4.92E+00. Sample per second: 15123\n",
      "MF_BPR: Epoch 294 of 1500. Elapsed time 59.82 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 4.98E+00. Sample per second: 14625\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.39 sec. Users per second: 1678\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.73 sec. Users per second: 1710\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2295257, PRECISION: 0.0782954, PRECISION_RECALL_MIN_DEN: 0.0989597, RECALL: 0.0644287, MAP: 0.0455640, MRR: 0.1887130, NDCG: 0.0640063, F1: 0.0706884, HIT_RATE: 0.7829540, ARHR: 0.2605934, NOVELTY: 0.0098341, AVERAGE_POPULARITY: 0.5303974, DIVERSITY_MEAN_INTER_LIST: 0.8634304, DIVERSITY_HERFINDAHL: 0.9863418, COVERAGE_ITEM: 0.9968168, COVERAGE_ITEM_CORRECT: 0.0550510, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4044334, DIVERSITY_GINI: 0.1686444, SHANNON_ENTROPY: 8.6018349, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 295 of 1500. Elapsed time 1.01 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 5.03E+00. Sample per second: 14561\n",
      "MF_BPR: Epoch 296 of 1500. Elapsed time 1.01 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 5.13E+00. Sample per second: 14072\n",
      "MF_BPR: Epoch 297 of 1500. Elapsed time 1.01 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.30 seconds. BPR loss 5.07E+00. Sample per second: 13191\n",
      "MF_BPR: Epoch 298 of 1500. Elapsed time 1.01 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 5.13E+00. Sample per second: 15357\n",
      "MF_BPR: Epoch 299 of 1500. Elapsed time 1.01 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 5.16E+00. Sample per second: 14716\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.06 sec. Users per second: 1697\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.52 sec. Users per second: 1718\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2299961, PRECISION: 0.0785194, PRECISION_RECALL_MIN_DEN: 0.0992056, RECALL: 0.0645184, MAP: 0.0456805, MRR: 0.1891881, NDCG: 0.0642129, F1: 0.0708337, HIT_RATE: 0.7851942, ARHR: 0.2612949, NOVELTY: 0.0098249, AVERAGE_POPULARITY: 0.5313898, DIVERSITY_MEAN_INTER_LIST: 0.8631594, DIVERSITY_HERFINDAHL: 0.9863147, COVERAGE_ITEM: 0.9969104, COVERAGE_ITEM_CORRECT: 0.0548638, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4052062, DIVERSITY_GINI: 0.1675010, SHANNON_ENTROPY: 8.5944024, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 300 of 1500. Elapsed time 1.03 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 5.09E+00. Sample per second: 15279\n",
      "MF_BPR: Epoch 301 of 1500. Elapsed time 1.03 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 5.23E+00. Sample per second: 14662\n",
      "MF_BPR: Epoch 302 of 1500. Elapsed time 1.03 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.98 seconds. BPR loss 5.22E+00. Sample per second: 14035\n",
      "MF_BPR: Epoch 303 of 1500. Elapsed time 1.03 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.22 seconds. BPR loss 5.34E+00. Sample per second: 13392\n",
      "MF_BPR: Epoch 304 of 1500. Elapsed time 1.03 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 5.34E+00. Sample per second: 15902\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.03 sec. Users per second: 1732\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.11 sec. Users per second: 1736\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2302524, PRECISION: 0.0787075, PRECISION_RECALL_MIN_DEN: 0.0993578, RECALL: 0.0645158, MAP: 0.0457476, MRR: 0.1894976, NDCG: 0.0643561, F1: 0.0709086, HIT_RATE: 0.7870755, ARHR: 0.2618410, NOVELTY: 0.0098166, AVERAGE_POPULARITY: 0.5322600, DIVERSITY_MEAN_INTER_LIST: 0.8630328, DIVERSITY_HERFINDAHL: 0.9863020, COVERAGE_ITEM: 0.9969104, COVERAGE_ITEM_CORRECT: 0.0547702, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4059790, DIVERSITY_GINI: 0.1664174, SHANNON_ENTROPY: 8.5881347, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 305 of 1500. Elapsed time 1.04 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 5.43E+00. Sample per second: 14349\n",
      "MF_BPR: Epoch 306 of 1500. Elapsed time 1.04 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 5.45E+00. Sample per second: 13560\n",
      "MF_BPR: Epoch 307 of 1500. Elapsed time 1.05 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 5.45E+00. Sample per second: 15996\n",
      "MF_BPR: Epoch 308 of 1500. Elapsed time 1.05 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 5.50E+00. Sample per second: 15185\n",
      "MF_BPR: Epoch 309 of 1500. Elapsed time 1.05 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 5.52E+00. Sample per second: 14539\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.31 sec. Users per second: 1749\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.67 sec. Users per second: 1756\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2305688, PRECISION: 0.0789359, PRECISION_RECALL_MIN_DEN: 0.0996118, RECALL: 0.0646421, MAP: 0.0458577, MRR: 0.1898892, NDCG: 0.0645649, F1: 0.0710775, HIT_RATE: 0.7893588, ARHR: 0.2624155, NOVELTY: 0.0098095, AVERAGE_POPULARITY: 0.5328969, DIVERSITY_MEAN_INTER_LIST: 0.8630669, DIVERSITY_HERFINDAHL: 0.9863055, COVERAGE_ITEM: 0.9966295, COVERAGE_ITEM_CORRECT: 0.0545829, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4069807, DIVERSITY_GINI: 0.1653519, SHANNON_ENTROPY: 8.5833326, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 310 of 1500. Elapsed time 1.06 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 5.59E+00. Sample per second: 14688\n",
      "MF_BPR: Epoch 311 of 1500. Elapsed time 1.06 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 5.54E+00. Sample per second: 14130\n",
      "MF_BPR: Epoch 312 of 1500. Elapsed time 1.06 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 5.61E+00. Sample per second: 13692\n",
      "MF_BPR: Epoch 313 of 1500. Elapsed time 1.06 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 5.66E+00. Sample per second: 15806\n",
      "MF_BPR: Epoch 314 of 1500. Elapsed time 1.07 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 5.66E+00. Sample per second: 14622\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.04 sec. Users per second: 1698\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.65 sec. Users per second: 1713\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2308764, PRECISION: 0.0791312, PRECISION_RECALL_MIN_DEN: 0.0998228, RECALL: 0.0646965, MAP: 0.0459455, MRR: 0.1901014, NDCG: 0.0647066, F1: 0.0711895, HIT_RATE: 0.7913118, ARHR: 0.2628355, NOVELTY: 0.0098025, AVERAGE_POPULARITY: 0.5335198, DIVERSITY_MEAN_INTER_LIST: 0.8630105, DIVERSITY_HERFINDAHL: 0.9862998, COVERAGE_ITEM: 0.9968168, COVERAGE_ITEM_CORRECT: 0.0543957, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4076390, DIVERSITY_GINI: 0.1643501, SHANNON_ENTROPY: 8.5788928, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 315 of 1500. Elapsed time 1.08 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 5.71E+00. Sample per second: 14630\n",
      "MF_BPR: Epoch 316 of 1500. Elapsed time 1.08 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 5.75E+00. Sample per second: 13995\n",
      "MF_BPR: Epoch 317 of 1500. Elapsed time 1.08 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.27 seconds. BPR loss 5.79E+00. Sample per second: 13254\n",
      "MF_BPR: Epoch 318 of 1500. Elapsed time 1.08 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 5.81E+00. Sample per second: 14946\n",
      "MF_BPR: Epoch 319 of 1500. Elapsed time 1.08 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 5.85E+00. Sample per second: 13742\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.52 sec. Users per second: 1704\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.87 sec. Users per second: 1704\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2311799, PRECISION: 0.0793251, PRECISION_RECALL_MIN_DEN: 0.1000559, RECALL: 0.0648047, MAP: 0.0459998, MRR: 0.1902726, NDCG: 0.0648709, F1: 0.0713334, HIT_RATE: 0.7932505, ARHR: 0.2631677, NOVELTY: 0.0097968, AVERAGE_POPULARITY: 0.5338890, DIVERSITY_MEAN_INTER_LIST: 0.8631453, DIVERSITY_HERFINDAHL: 0.9863133, COVERAGE_ITEM: 0.9967232, COVERAGE_ITEM_CORRECT: 0.0542084, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4086694, DIVERSITY_GINI: 0.1635005, SHANNON_ENTROPY: 8.5765175, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 320 of 1500. Elapsed time 1.10 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.26 seconds. BPR loss 5.94E+00. Sample per second: 16391\n",
      "MF_BPR: Epoch 321 of 1500. Elapsed time 1.10 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 5.93E+00. Sample per second: 15808\n",
      "MF_BPR: Epoch 322 of 1500. Elapsed time 1.10 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 5.99E+00. Sample per second: 15158\n",
      "MF_BPR: Epoch 323 of 1500. Elapsed time 1.10 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 6.00E+00. Sample per second: 14487\n",
      "MF_BPR: Epoch 324 of 1500. Elapsed time 1.10 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 6.09E+00. Sample per second: 13980\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.44 sec. Users per second: 1708\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.38 sec. Users per second: 1725\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2313531, PRECISION: 0.0794801, PRECISION_RECALL_MIN_DEN: 0.1002004, RECALL: 0.0648329, MAP: 0.0460507, MRR: 0.1903762, NDCG: 0.0649620, F1: 0.0714132, HIT_RATE: 0.7948015, ARHR: 0.2634787, NOVELTY: 0.0097914, AVERAGE_POPULARITY: 0.5343310, DIVERSITY_MEAN_INTER_LIST: 0.8631024, DIVERSITY_HERFINDAHL: 0.9863090, COVERAGE_ITEM: 0.9968168, COVERAGE_ITEM_CORRECT: 0.0540212, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4089985, DIVERSITY_GINI: 0.1625884, SHANNON_ENTROPY: 8.5734849, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 325 of 1500. Elapsed time 1.11 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 6.07E+00. Sample per second: 15056\n",
      "MF_BPR: Epoch 326 of 1500. Elapsed time 1.11 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 6.16E+00. Sample per second: 14627\n",
      "MF_BPR: Epoch 327 of 1500. Elapsed time 1.11 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 6.19E+00. Sample per second: 14206\n",
      "MF_BPR: Epoch 328 of 1500. Elapsed time 1.12 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 6.14E+00. Sample per second: 13866\n",
      "MF_BPR: Epoch 329 of 1500. Elapsed time 1.12 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.16 seconds. BPR loss 6.20E+00. Sample per second: 16796\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.19 sec. Users per second: 1822\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.71 sec. Users per second: 1847\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2316153, PRECISION: 0.0796252, PRECISION_RECALL_MIN_DEN: 0.1004180, RECALL: 0.0649817, MAP: 0.0461477, MRR: 0.1906058, NDCG: 0.0651018, F1: 0.0715620, HIT_RATE: 0.7962519, ARHR: 0.2638945, NOVELTY: 0.0097867, AVERAGE_POPULARITY: 0.5343338, DIVERSITY_MEAN_INTER_LIST: 0.8636614, DIVERSITY_HERFINDAHL: 0.9863649, COVERAGE_ITEM: 0.9966295, COVERAGE_ITEM_CORRECT: 0.0537403, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4095137, DIVERSITY_GINI: 0.1618295, SHANNON_ENTROPY: 8.5734199, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 330 of 1500. Elapsed time 1.13 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.36 seconds. BPR loss 6.31E+00. Sample per second: 13034\n",
      "MF_BPR: Epoch 331 of 1500. Elapsed time 1.13 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.73 seconds. BPR loss 6.38E+00. Sample per second: 14767\n",
      "MF_BPR: Epoch 332 of 1500. Elapsed time 1.13 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 6.34E+00. Sample per second: 14289\n",
      "MF_BPR: Epoch 333 of 1500. Elapsed time 1.13 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 6.38E+00. Sample per second: 13864\n",
      "MF_BPR: Epoch 334 of 1500. Elapsed time 1.13 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.15 seconds. BPR loss 6.42E+00. Sample per second: 16825\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.09 sec. Users per second: 1861\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.58 sec. Users per second: 1853\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2318942, PRECISION: 0.0797516, PRECISION_RECALL_MIN_DEN: 0.1005254, RECALL: 0.0649954, MAP: 0.0461615, MRR: 0.1905893, NDCG: 0.0651870, F1: 0.0716213, HIT_RATE: 0.7975156, ARHR: 0.2640072, NOVELTY: 0.0097825, AVERAGE_POPULARITY: 0.5344722, DIVERSITY_MEAN_INTER_LIST: 0.8639941, DIVERSITY_HERFINDAHL: 0.9863982, COVERAGE_ITEM: 0.9966295, COVERAGE_ITEM_CORRECT: 0.0538339, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4103151, DIVERSITY_GINI: 0.1611915, SHANNON_ENTROPY: 8.5740397, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 335 of 1500. Elapsed time 1.14 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 6.43E+00. Sample per second: 13985\n",
      "MF_BPR: Epoch 336 of 1500. Elapsed time 1.15 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.14 seconds. BPR loss 6.50E+00. Sample per second: 13599\n",
      "MF_BPR: Epoch 337 of 1500. Elapsed time 1.15 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 6.55E+00. Sample per second: 16216\n",
      "MF_BPR: Epoch 338 of 1500. Elapsed time 1.15 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 6.61E+00. Sample per second: 15661\n",
      "MF_BPR: Epoch 339 of 1500. Elapsed time 1.15 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 6.64E+00. Sample per second: 15017\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.48 sec. Users per second: 1772\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.66 sec. Users per second: 1801\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2321361, PRECISION: 0.0798980, PRECISION_RECALL_MIN_DEN: 0.1006700, RECALL: 0.0650522, MAP: 0.0462472, MRR: 0.1910266, NDCG: 0.0653039, F1: 0.0717149, HIT_RATE: 0.7989804, ARHR: 0.2645380, NOVELTY: 0.0097784, AVERAGE_POPULARITY: 0.5345832, DIVERSITY_MEAN_INTER_LIST: 0.8643974, DIVERSITY_HERFINDAHL: 0.9864385, COVERAGE_ITEM: 0.9965359, COVERAGE_ITEM_CORRECT: 0.0536467, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4108875, DIVERSITY_GINI: 0.1604679, SHANNON_ENTROPY: 8.5738994, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 340 of 1500. Elapsed time 1.16 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 6.70E+00. Sample per second: 15330\n",
      "MF_BPR: Epoch 341 of 1500. Elapsed time 1.16 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 6.63E+00. Sample per second: 14975\n",
      "MF_BPR: Epoch 342 of 1500. Elapsed time 1.16 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 6.67E+00. Sample per second: 14559\n",
      "MF_BPR: Epoch 343 of 1500. Elapsed time 1.16 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.96 seconds. BPR loss 6.67E+00. Sample per second: 14098\n",
      "MF_BPR: Epoch 344 of 1500. Elapsed time 1.17 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 6.68E+00. Sample per second: 14107\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 57000 ( 81.86% ) in 30.44 sec. Users per second: 1872\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 36.96 sec. Users per second: 1884\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2322148, PRECISION: 0.0800072, PRECISION_RECALL_MIN_DEN: 0.1007964, RECALL: 0.0651011, MAP: 0.0463150, MRR: 0.1912081, NDCG: 0.0654100, F1: 0.0717885, HIT_RATE: 0.8000718, ARHR: 0.2648726, NOVELTY: 0.0097746, AVERAGE_POPULARITY: 0.5346301, DIVERSITY_MEAN_INTER_LIST: 0.8647190, DIVERSITY_HERFINDAHL: 0.9864707, COVERAGE_ITEM: 0.9965359, COVERAGE_ITEM_CORRECT: 0.0528977, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4112310, DIVERSITY_GINI: 0.1597802, SHANNON_ENTROPY: 8.5734484, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 345 of 1500. Elapsed time 1.18 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.11 seconds. BPR loss 6.79E+00. Sample per second: 17021\n",
      "MF_BPR: Epoch 346 of 1500. Elapsed time 1.18 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 6.84E+00. Sample per second: 16257\n",
      "MF_BPR: Epoch 347 of 1500. Elapsed time 1.18 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 6.84E+00. Sample per second: 15819\n",
      "MF_BPR: Epoch 348 of 1500. Elapsed time 1.18 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 6.86E+00. Sample per second: 15421\n",
      "MF_BPR: Epoch 349 of 1500. Elapsed time 1.18 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 6.90E+00. Sample per second: 15044\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.02 sec. Users per second: 1832\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.69 sec. Users per second: 1848\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2322932, PRECISION: 0.0801522, PRECISION_RECALL_MIN_DEN: 0.1009623, RECALL: 0.0651821, MAP: 0.0463330, MRR: 0.1910593, NDCG: 0.0654693, F1: 0.0718962, HIT_RATE: 0.8015222, ARHR: 0.2648646, NOVELTY: 0.0097711, AVERAGE_POPULARITY: 0.5346490, DIVERSITY_MEAN_INTER_LIST: 0.8650280, DIVERSITY_HERFINDAHL: 0.9865016, COVERAGE_ITEM: 0.9961614, COVERAGE_ITEM_CORRECT: 0.0534594, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4117462, DIVERSITY_GINI: 0.1591213, SHANNON_ENTROPY: 8.5732940, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 350 of 1500. Elapsed time 1.19 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 6.98E+00. Sample per second: 15300\n",
      "MF_BPR: Epoch 351 of 1500. Elapsed time 1.19 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 6.97E+00. Sample per second: 15002\n",
      "MF_BPR: Epoch 352 of 1500. Elapsed time 1.20 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 7.05E+00. Sample per second: 14645\n",
      "MF_BPR: Epoch 353 of 1500. Elapsed time 1.20 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 7.06E+00. Sample per second: 14209\n",
      "MF_BPR: Epoch 354 of 1500. Elapsed time 1.20 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 7.07E+00. Sample per second: 13689\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.12 sec. Users per second: 1826\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.94 sec. Users per second: 1835\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2323830, PRECISION: 0.0802441, PRECISION_RECALL_MIN_DEN: 0.1010333, RECALL: 0.0651953, MAP: 0.0463661, MRR: 0.1911119, NDCG: 0.0655140, F1: 0.0719412, HIT_RATE: 0.8024413, ARHR: 0.2650426, NOVELTY: 0.0097685, AVERAGE_POPULARITY: 0.5344678, DIVERSITY_MEAN_INTER_LIST: 0.8655394, DIVERSITY_HERFINDAHL: 0.9865527, COVERAGE_ITEM: 0.9964423, COVERAGE_ITEM_CORRECT: 0.0530849, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4119179, DIVERSITY_GINI: 0.1584189, SHANNON_ENTROPY: 8.5744849, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 355 of 1500. Elapsed time 1.21 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 7.10E+00. Sample per second: 16539\n",
      "MF_BPR: Epoch 356 of 1500. Elapsed time 1.21 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 7.11E+00. Sample per second: 16151\n",
      "MF_BPR: Epoch 357 of 1500. Elapsed time 1.21 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 7.22E+00. Sample per second: 15549\n",
      "MF_BPR: Epoch 358 of 1500. Elapsed time 1.21 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 7.19E+00. Sample per second: 14538\n",
      "MF_BPR: Epoch 359 of 1500. Elapsed time 1.21 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.38 seconds. BPR loss 7.23E+00. Sample per second: 12991\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.30 sec. Users per second: 1749\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.80 sec. Users per second: 1750\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2322977, PRECISION: 0.0803475, PRECISION_RECALL_MIN_DEN: 0.1011369, RECALL: 0.0652424, MAP: 0.0464444, MRR: 0.1912704, NDCG: 0.0656530, F1: 0.0720114, HIT_RATE: 0.8034753, ARHR: 0.2653097, NOVELTY: 0.0097655, AVERAGE_POPULARITY: 0.5342246, DIVERSITY_MEAN_INTER_LIST: 0.8661360, DIVERSITY_HERFINDAHL: 0.9866124, COVERAGE_ITEM: 0.9965359, COVERAGE_ITEM_CORRECT: 0.0532722, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4122041, DIVERSITY_GINI: 0.1578124, SHANNON_ENTROPY: 8.5767388, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 360 of 1500. Elapsed time 1.23 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 7.26E+00. Sample per second: 14259\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 361 of 1500. Elapsed time 1.23 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.20 seconds. BPR loss 7.31E+00. Sample per second: 13431\n",
      "MF_BPR: Epoch 362 of 1500. Elapsed time 1.23 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 7.43E+00. Sample per second: 15228\n",
      "MF_BPR: Epoch 363 of 1500. Elapsed time 1.23 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.62 seconds. BPR loss 7.31E+00. Sample per second: 12439\n",
      "MF_BPR: Epoch 364 of 1500. Elapsed time 1.23 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.35 seconds. BPR loss 7.43E+00. Sample per second: 13071\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.20 sec. Users per second: 1722\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.43 sec. Users per second: 1722\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2324574, PRECISION: 0.0804452, PRECISION_RECALL_MIN_DEN: 0.1012777, RECALL: 0.0653190, MAP: 0.0464696, MRR: 0.1911523, NDCG: 0.0657304, F1: 0.0720972, HIT_RATE: 0.8044518, ARHR: 0.2653320, NOVELTY: 0.0097635, AVERAGE_POPULARITY: 0.5339113, DIVERSITY_MEAN_INTER_LIST: 0.8668248, DIVERSITY_HERFINDAHL: 0.9866812, COVERAGE_ITEM: 0.9963487, COVERAGE_ITEM_CORRECT: 0.0530849, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4122900, DIVERSITY_GINI: 0.1572509, SHANNON_ENTROPY: 8.5802875, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 365 of 1500. Elapsed time 1.24 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 7.45E+00. Sample per second: 13705\n",
      "MF_BPR: Epoch 366 of 1500. Elapsed time 1.25 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 7.50E+00. Sample per second: 16320\n",
      "MF_BPR: Epoch 367 of 1500. Elapsed time 1.25 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 7.53E+00. Sample per second: 15495\n",
      "MF_BPR: Epoch 368 of 1500. Elapsed time 1.25 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.21 seconds. BPR loss 7.42E+00. Sample per second: 13403\n",
      "MF_BPR: Epoch 369 of 1500. Elapsed time 1.25 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 7.56E+00. Sample per second: 14219\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.31 sec. Users per second: 1716\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.29 sec. Users per second: 1728\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2323927, PRECISION: 0.0805141, PRECISION_RECALL_MIN_DEN: 0.1013164, RECALL: 0.0652774, MAP: 0.0465200, MRR: 0.1912802, NDCG: 0.0657745, F1: 0.0720996, HIT_RATE: 0.8051411, ARHR: 0.2655373, NOVELTY: 0.0097617, AVERAGE_POPULARITY: 0.5336619, DIVERSITY_MEAN_INTER_LIST: 0.8671918, DIVERSITY_HERFINDAHL: 0.9867179, COVERAGE_ITEM: 0.9965359, COVERAGE_ITEM_CORRECT: 0.0531785, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4123902, DIVERSITY_GINI: 0.1567837, SHANNON_ENTROPY: 8.5819999, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 370 of 1500. Elapsed time 1.26 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 7.59E+00. Sample per second: 14983\n",
      "MF_BPR: Epoch 371 of 1500. Elapsed time 1.26 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 7.63E+00. Sample per second: 13778\n",
      "MF_BPR: Epoch 372 of 1500. Elapsed time 1.26 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 7.70E+00. Sample per second: 14932\n",
      "MF_BPR: Epoch 373 of 1500. Elapsed time 1.27 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.27 seconds. BPR loss 7.66E+00. Sample per second: 13270\n",
      "MF_BPR: Epoch 374 of 1500. Elapsed time 1.27 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 7.72E+00. Sample per second: 15120\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.39 sec. Users per second: 1744\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.89 sec. Users per second: 1746\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2323734, PRECISION: 0.0806204, PRECISION_RECALL_MIN_DEN: 0.1014929, RECALL: 0.0653959, MAP: 0.0465763, MRR: 0.1913659, NDCG: 0.0658993, F1: 0.0722144, HIT_RATE: 0.8062038, ARHR: 0.2657548, NOVELTY: 0.0097601, AVERAGE_POPULARITY: 0.5333025, DIVERSITY_MEAN_INTER_LIST: 0.8678439, DIVERSITY_HERFINDAHL: 0.9867831, COVERAGE_ITEM: 0.9963487, COVERAGE_ITEM_CORRECT: 0.0532722, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4126191, DIVERSITY_GINI: 0.1562766, SHANNON_ENTROPY: 8.5853287, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 375 of 1500. Elapsed time 1.28 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 7.78E+00. Sample per second: 14478\n",
      "MF_BPR: Epoch 376 of 1500. Elapsed time 1.28 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 7.80E+00. Sample per second: 13872\n",
      "MF_BPR: Epoch 377 of 1500. Elapsed time 1.28 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.20 seconds. BPR loss 7.82E+00. Sample per second: 16650\n",
      "MF_BPR: Epoch 378 of 1500. Elapsed time 1.28 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 7.76E+00. Sample per second: 15875\n",
      "MF_BPR: Epoch 379 of 1500. Elapsed time 1.28 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 7.85E+00. Sample per second: 14176\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.34 sec. Users per second: 1747\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.79 sec. Users per second: 1750\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2323816, PRECISION: 0.0807037, PRECISION_RECALL_MIN_DEN: 0.1016079, RECALL: 0.0654786, MAP: 0.0466080, MRR: 0.1914288, NDCG: 0.0659776, F1: 0.0722983, HIT_RATE: 0.8070367, ARHR: 0.2658964, NOVELTY: 0.0097591, AVERAGE_POPULARITY: 0.5328213, DIVERSITY_MEAN_INTER_LIST: 0.8685236, DIVERSITY_HERFINDAHL: 0.9868511, COVERAGE_ITEM: 0.9963487, COVERAGE_ITEM_CORRECT: 0.0528977, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4130628, DIVERSITY_GINI: 0.1557290, SHANNON_ENTROPY: 8.5889497, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 380 of 1500. Elapsed time 1.30 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 7.86E+00. Sample per second: 13940\n",
      "MF_BPR: Epoch 381 of 1500. Elapsed time 1.30 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 7.89E+00. Sample per second: 15889\n",
      "MF_BPR: Epoch 382 of 1500. Elapsed time 1.30 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 7.91E+00. Sample per second: 15161\n",
      "MF_BPR: Epoch 383 of 1500. Elapsed time 1.30 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 8.03E+00. Sample per second: 14123\n",
      "MF_BPR: Epoch 384 of 1500. Elapsed time 1.30 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.23 seconds. BPR loss 8.01E+00. Sample per second: 13369\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.53 sec. Users per second: 1769\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.14 sec. Users per second: 1779\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2324317, PRECISION: 0.0808099, PRECISION_RECALL_MIN_DEN: 0.1017344, RECALL: 0.0655292, MAP: 0.0466641, MRR: 0.1915211, NDCG: 0.0660394, F1: 0.0723717, HIT_RATE: 0.8080994, ARHR: 0.2661710, NOVELTY: 0.0097572, AVERAGE_POPULARITY: 0.5325141, DIVERSITY_MEAN_INTER_LIST: 0.8690195, DIVERSITY_HERFINDAHL: 0.9869007, COVERAGE_ITEM: 0.9964423, COVERAGE_ITEM_CORRECT: 0.0528040, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4130055, DIVERSITY_GINI: 0.1552602, SHANNON_ENTROPY: 8.5921361, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 385 of 1500. Elapsed time 1.31 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 8.10E+00. Sample per second: 14898\n",
      "MF_BPR: Epoch 386 of 1500. Elapsed time 1.31 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 8.13E+00. Sample per second: 13774\n",
      "MF_BPR: Epoch 387 of 1500. Elapsed time 1.32 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.26 seconds. BPR loss 8.17E+00. Sample per second: 16401\n",
      "MF_BPR: Epoch 388 of 1500. Elapsed time 1.32 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 8.18E+00. Sample per second: 15946\n",
      "MF_BPR: Epoch 389 of 1500. Elapsed time 1.32 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 8.17E+00. Sample per second: 14974\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.30 sec. Users per second: 1782\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.94 sec. Users per second: 1788\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2324984, PRECISION: 0.0809047, PRECISION_RECALL_MIN_DEN: 0.1017819, RECALL: 0.0655148, MAP: 0.0466838, MRR: 0.1915603, NDCG: 0.0660543, F1: 0.0724009, HIT_RATE: 0.8090472, ARHR: 0.2662713, NOVELTY: 0.0097561, AVERAGE_POPULARITY: 0.5319907, DIVERSITY_MEAN_INTER_LIST: 0.8697643, DIVERSITY_HERFINDAHL: 0.9869752, COVERAGE_ITEM: 0.9964423, COVERAGE_ITEM_CORRECT: 0.0524295, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4134348, DIVERSITY_GINI: 0.1547760, SHANNON_ENTROPY: 8.5962792, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 390 of 1500. Elapsed time 1.33 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 8.11E+00. Sample per second: 14424\n",
      "MF_BPR: Epoch 391 of 1500. Elapsed time 1.33 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 8.19E+00. Sample per second: 14117\n",
      "MF_BPR: Epoch 392 of 1500. Elapsed time 1.33 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 8.21E+00. Sample per second: 13881\n",
      "MF_BPR: Epoch 393 of 1500. Elapsed time 1.33 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 8.26E+00. Sample per second: 16565\n",
      "MF_BPR: Epoch 394 of 1500. Elapsed time 1.33 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 8.35E+00. Sample per second: 15859\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.07 sec. Users per second: 1762\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.33 sec. Users per second: 1771\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2324180, PRECISION: 0.0810239, PRECISION_RECALL_MIN_DEN: 0.1019115, RECALL: 0.0655973, MAP: 0.0467216, MRR: 0.1915592, NDCG: 0.0661163, F1: 0.0724990, HIT_RATE: 0.8102391, ARHR: 0.2664130, NOVELTY: 0.0097552, AVERAGE_POPULARITY: 0.5315719, DIVERSITY_MEAN_INTER_LIST: 0.8702994, DIVERSITY_HERFINDAHL: 0.9870287, COVERAGE_ITEM: 0.9963487, COVERAGE_ITEM_CORRECT: 0.0528040, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4137354, DIVERSITY_GINI: 0.1543170, SHANNON_ENTROPY: 8.6000563, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 395 of 1500. Elapsed time 1.35 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.20 seconds. BPR loss 8.30E+00. Sample per second: 13445\n",
      "MF_BPR: Epoch 396 of 1500. Elapsed time 1.35 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 8.38E+00. Sample per second: 15082\n",
      "MF_BPR: Epoch 397 of 1500. Elapsed time 1.35 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 8.47E+00. Sample per second: 14175\n",
      "MF_BPR: Epoch 398 of 1500. Elapsed time 1.35 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 8.48E+00. Sample per second: 13655\n",
      "MF_BPR: Epoch 399 of 1500. Elapsed time 1.35 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 8.44E+00. Sample per second: 15908\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.04 sec. Users per second: 1798\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.53 sec. Users per second: 1807\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2324015, PRECISION: 0.0810598, PRECISION_RECALL_MIN_DEN: 0.1019559, RECALL: 0.0656313, MAP: 0.0467376, MRR: 0.1915770, NDCG: 0.0661633, F1: 0.0725342, HIT_RATE: 0.8105981, ARHR: 0.2664816, NOVELTY: 0.0097540, AVERAGE_POPULARITY: 0.5310773, DIVERSITY_MEAN_INTER_LIST: 0.8709993, DIVERSITY_HERFINDAHL: 0.9870987, COVERAGE_ITEM: 0.9961614, COVERAGE_ITEM_CORRECT: 0.0528977, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4137354, DIVERSITY_GINI: 0.1539377, SHANNON_ENTROPY: 8.6052222, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 400 of 1500. Elapsed time 1.36 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 8.55E+00. Sample per second: 16311\n",
      "MF_BPR: Epoch 401 of 1500. Elapsed time 1.36 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 8.43E+00. Sample per second: 15588\n",
      "MF_BPR: Epoch 402 of 1500. Elapsed time 1.37 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.73 seconds. BPR loss 8.51E+00. Sample per second: 14767\n",
      "MF_BPR: Epoch 403 of 1500. Elapsed time 1.37 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.96 seconds. BPR loss 8.67E+00. Sample per second: 14083\n",
      "MF_BPR: Epoch 404 of 1500. Elapsed time 1.37 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.36 seconds. BPR loss 8.58E+00. Sample per second: 13029\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.22 sec. Users per second: 1754\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.96 sec. Users per second: 1743\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2323260, PRECISION: 0.0811287, PRECISION_RECALL_MIN_DEN: 0.1020652, RECALL: 0.0657082, MAP: 0.0467497, MRR: 0.1915746, NDCG: 0.0662427, F1: 0.0726087, HIT_RATE: 0.8112874, ARHR: 0.2664921, NOVELTY: 0.0097526, AVERAGE_POPULARITY: 0.5306297, DIVERSITY_MEAN_INTER_LIST: 0.8716078, DIVERSITY_HERFINDAHL: 0.9871595, COVERAGE_ITEM: 0.9960678, COVERAGE_ITEM_CORRECT: 0.0536467, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4142506, DIVERSITY_GINI: 0.1534551, SHANNON_ENTROPY: 8.6094735, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 405 of 1500. Elapsed time 1.38 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 8.66E+00. Sample per second: 14974\n",
      "MF_BPR: Epoch 406 of 1500. Elapsed time 1.38 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 8.56E+00. Sample per second: 14320\n",
      "MF_BPR: Epoch 407 of 1500. Elapsed time 1.38 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 8.66E+00. Sample per second: 13701\n",
      "MF_BPR: Epoch 408 of 1500. Elapsed time 1.38 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 8.71E+00. Sample per second: 15943\n",
      "MF_BPR: Epoch 409 of 1500. Elapsed time 1.38 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 8.70E+00. Sample per second: 15291\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.01 sec. Users per second: 1699\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.32 sec. Users per second: 1727\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2322473, PRECISION: 0.0811388, PRECISION_RECALL_MIN_DEN: 0.1020741, RECALL: 0.0657104, MAP: 0.0467612, MRR: 0.1915609, NDCG: 0.0663008, F1: 0.0726141, HIT_RATE: 0.8113880, ARHR: 0.2664904, NOVELTY: 0.0097525, AVERAGE_POPULARITY: 0.5299084, DIVERSITY_MEAN_INTER_LIST: 0.8723626, DIVERSITY_HERFINDAHL: 0.9872350, COVERAGE_ITEM: 0.9963487, COVERAGE_ITEM_CORRECT: 0.0536467, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4142649, DIVERSITY_GINI: 0.1530954, SHANNON_ENTROPY: 8.6150305, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 410 of 1500. Elapsed time 1.40 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 8.77E+00. Sample per second: 13562\n",
      "MF_BPR: Epoch 411 of 1500. Elapsed time 1.40 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 8.86E+00. Sample per second: 16230\n",
      "MF_BPR: Epoch 412 of 1500. Elapsed time 1.40 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 8.76E+00. Sample per second: 15627\n",
      "MF_BPR: Epoch 413 of 1500. Elapsed time 1.40 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 8.84E+00. Sample per second: 15047\n",
      "MF_BPR: Epoch 414 of 1500. Elapsed time 1.40 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 8.83E+00. Sample per second: 14571\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.20 sec. Users per second: 1755\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.51 sec. Users per second: 1763\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2322645, PRECISION: 0.0812106, PRECISION_RECALL_MIN_DEN: 0.1021718, RECALL: 0.0657822, MAP: 0.0468488, MRR: 0.1917947, NDCG: 0.0663891, F1: 0.0726867, HIT_RATE: 0.8121060, ARHR: 0.2668107, NOVELTY: 0.0097520, AVERAGE_POPULARITY: 0.5292989, DIVERSITY_MEAN_INTER_LIST: 0.8731053, DIVERSITY_HERFINDAHL: 0.9873093, COVERAGE_ITEM: 0.9960678, COVERAGE_ITEM_CORRECT: 0.0535530, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4141218, DIVERSITY_GINI: 0.1526160, SHANNON_ENTROPY: 8.6197108, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 415 of 1500. Elapsed time 1.41 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 8.90E+00. Sample per second: 15151\n",
      "MF_BPR: Epoch 416 of 1500. Elapsed time 1.42 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 8.89E+00. Sample per second: 14603\n",
      "MF_BPR: Epoch 417 of 1500. Elapsed time 1.42 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 8.94E+00. Sample per second: 13810\n",
      "MF_BPR: Epoch 418 of 1500. Elapsed time 1.42 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 9.03E+00. Sample per second: 15120\n",
      "MF_BPR: Epoch 419 of 1500. Elapsed time 1.42 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 9.03E+00. Sample per second: 14496\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.18 sec. Users per second: 1723\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.55 sec. Users per second: 1717\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2324540, PRECISION: 0.0813140, PRECISION_RECALL_MIN_DEN: 0.1022787, RECALL: 0.0658372, MAP: 0.0468661, MRR: 0.1919078, NDCG: 0.0664463, F1: 0.0727617, HIT_RATE: 0.8131399, ARHR: 0.2669540, NOVELTY: 0.0097510, AVERAGE_POPULARITY: 0.5287768, DIVERSITY_MEAN_INTER_LIST: 0.8737636, DIVERSITY_HERFINDAHL: 0.9873751, COVERAGE_ITEM: 0.9959742, COVERAGE_ITEM_CORRECT: 0.0537403, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4148516, DIVERSITY_GINI: 0.1521773, SHANNON_ENTROPY: 8.6240726, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 420 of 1500. Elapsed time 1.43 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 9.10E+00. Sample per second: 14825\n",
      "MF_BPR: Epoch 421 of 1500. Elapsed time 1.43 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 9.08E+00. Sample per second: 14205\n",
      "MF_BPR: Epoch 422 of 1500. Elapsed time 1.43 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.20 seconds. BPR loss 9.10E+00. Sample per second: 13443\n",
      "MF_BPR: Epoch 423 of 1500. Elapsed time 1.43 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 9.09E+00. Sample per second: 15694\n",
      "MF_BPR: Epoch 424 of 1500. Elapsed time 1.44 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 9.18E+00. Sample per second: 14948\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.52 sec. Users per second: 1704\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.70 sec. Users per second: 1711\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2323378, PRECISION: 0.0813887, PRECISION_RECALL_MIN_DEN: 0.1024342, RECALL: 0.0659910, MAP: 0.0469307, MRR: 0.1919445, NDCG: 0.0664716, F1: 0.0728855, HIT_RATE: 0.8138867, ARHR: 0.2671298, NOVELTY: 0.0097512, AVERAGE_POPULARITY: 0.5278856, DIVERSITY_MEAN_INTER_LIST: 0.8748381, DIVERSITY_HERFINDAHL: 0.9874826, COVERAGE_ITEM: 0.9957869, COVERAGE_ITEM_CORRECT: 0.0539275, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4149375, DIVERSITY_GINI: 0.1517416, SHANNON_ENTROPY: 8.6309935, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 425 of 1500. Elapsed time 1.45 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 9.20E+00. Sample per second: 14729\n",
      "MF_BPR: Epoch 426 of 1500. Elapsed time 1.45 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 9.16E+00. Sample per second: 14122\n",
      "MF_BPR: Epoch 427 of 1500. Elapsed time 1.45 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.13 seconds. BPR loss 9.13E+00. Sample per second: 13612\n",
      "MF_BPR: Epoch 428 of 1500. Elapsed time 1.45 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 9.23E+00. Sample per second: 16092\n",
      "MF_BPR: Epoch 429 of 1500. Elapsed time 1.45 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 9.30E+00. Sample per second: 15407\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.04 sec. Users per second: 1731\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.03 sec. Users per second: 1740\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2324626, PRECISION: 0.0814432, PRECISION_RECALL_MIN_DEN: 0.1025820, RECALL: 0.0661382, MAP: 0.0469620, MRR: 0.1920445, NDCG: 0.0665817, F1: 0.0729971, HIT_RATE: 0.8144324, ARHR: 0.2672900, NOVELTY: 0.0097513, AVERAGE_POPULARITY: 0.5271748, DIVERSITY_MEAN_INTER_LIST: 0.8757472, DIVERSITY_HERFINDAHL: 0.9875735, COVERAGE_ITEM: 0.9955997, COVERAGE_ITEM_CORRECT: 0.0531785, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4152237, DIVERSITY_GINI: 0.1513705, SHANNON_ENTROPY: 8.6377585, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 430 of 1500. Elapsed time 1.47 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 9.29E+00. Sample per second: 14348\n",
      "MF_BPR: Epoch 431 of 1500. Elapsed time 1.47 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 9.34E+00. Sample per second: 13852\n",
      "MF_BPR: Epoch 432 of 1500. Elapsed time 1.47 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 9.41E+00. Sample per second: 16472\n",
      "MF_BPR: Epoch 433 of 1500. Elapsed time 1.47 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 9.36E+00. Sample per second: 15633\n",
      "MF_BPR: Epoch 434 of 1500. Elapsed time 1.47 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 9.40E+00. Sample per second: 14830\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.51 sec. Users per second: 1704\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.44 sec. Users per second: 1722\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2322709, PRECISION: 0.0815509, PRECISION_RECALL_MIN_DEN: 0.1027287, RECALL: 0.0662209, MAP: 0.0470216, MRR: 0.1921839, NDCG: 0.0666838, F1: 0.0730907, HIT_RATE: 0.8155094, ARHR: 0.2675307, NOVELTY: 0.0097511, AVERAGE_POPULARITY: 0.5266154, DIVERSITY_MEAN_INTER_LIST: 0.8763212, DIVERSITY_HERFINDAHL: 0.9876309, COVERAGE_ITEM: 0.9957869, COVERAGE_ITEM_CORRECT: 0.0534594, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4153382, DIVERSITY_GINI: 0.1509247, SHANNON_ENTROPY: 8.6416897, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 435 of 1500. Elapsed time 1.48 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 9.50E+00. Sample per second: 15543\n",
      "MF_BPR: Epoch 436 of 1500. Elapsed time 1.48 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 9.46E+00. Sample per second: 14611\n",
      "MF_BPR: Epoch 437 of 1500. Elapsed time 1.49 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 9.51E+00. Sample per second: 14106\n",
      "MF_BPR: Epoch 438 of 1500. Elapsed time 1.49 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.13 seconds. BPR loss 9.55E+00. Sample per second: 13634\n",
      "MF_BPR: Epoch 439 of 1500. Elapsed time 1.49 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 9.53E+00. Sample per second: 16091\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.17 sec. Users per second: 1691\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.82 sec. Users per second: 1706\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2322700, PRECISION: 0.0816055, PRECISION_RECALL_MIN_DEN: 0.1027756, RECALL: 0.0662424, MAP: 0.0470186, MRR: 0.1920712, NDCG: 0.0666867, F1: 0.0731258, HIT_RATE: 0.8160551, ARHR: 0.2674388, NOVELTY: 0.0097506, AVERAGE_POPULARITY: 0.5260485, DIVERSITY_MEAN_INTER_LIST: 0.8769967, DIVERSITY_HERFINDAHL: 0.9876984, COVERAGE_ITEM: 0.9958805, COVERAGE_ITEM_CORRECT: 0.0539275, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4156959, DIVERSITY_GINI: 0.1506036, SHANNON_ENTROPY: 8.6467047, \n",
      "\n",
      "MF_BPR: Epoch 440 of 1500. Elapsed time 1.50 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 9.51E+00. Sample per second: 15648\n",
      "MF_BPR: Epoch 441 of 1500. Elapsed time 1.50 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 9.60E+00. Sample per second: 15120\n",
      "MF_BPR: Epoch 442 of 1500. Elapsed time 1.50 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 9.61E+00. Sample per second: 14607\n",
      "MF_BPR: Epoch 443 of 1500. Elapsed time 1.50 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 9.59E+00. Sample per second: 13954\n",
      "MF_BPR: Epoch 444 of 1500. Elapsed time 1.50 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 9.57E+00. Sample per second: 16598\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.20 sec. Users per second: 1656\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.88 sec. Users per second: 1663\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2322643, PRECISION: 0.0817046, PRECISION_RECALL_MIN_DEN: 0.1029084, RECALL: 0.0663405, MAP: 0.0470842, MRR: 0.1921934, NDCG: 0.0667716, F1: 0.0732253, HIT_RATE: 0.8170460, ARHR: 0.2676776, NOVELTY: 0.0097510, AVERAGE_POPULARITY: 0.5254740, DIVERSITY_MEAN_INTER_LIST: 0.8776930, DIVERSITY_HERFINDAHL: 0.9877680, COVERAGE_ITEM: 0.9958805, COVERAGE_ITEM_CORRECT: 0.0539275, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4157818, DIVERSITY_GINI: 0.1502360, SHANNON_ENTROPY: 8.6515851, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 445 of 1500. Elapsed time 1.52 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 9.62E+00. Sample per second: 15592\n",
      "MF_BPR: Epoch 446 of 1500. Elapsed time 1.52 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 9.77E+00. Sample per second: 13549\n",
      "MF_BPR: Epoch 447 of 1500. Elapsed time 1.52 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 9.67E+00. Sample per second: 15614\n",
      "MF_BPR: Epoch 448 of 1500. Elapsed time 1.52 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 9.70E+00. Sample per second: 14894\n",
      "MF_BPR: Epoch 449 of 1500. Elapsed time 1.52 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 9.77E+00. Sample per second: 14235\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.42 sec. Users per second: 1644\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.78 sec. Users per second: 1667\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2322093, PRECISION: 0.0817965, PRECISION_RECALL_MIN_DEN: 0.1029999, RECALL: 0.0663648, MAP: 0.0471686, MRR: 0.1923690, NDCG: 0.0668343, F1: 0.0732770, HIT_RATE: 0.8179651, ARHR: 0.2679820, NOVELTY: 0.0097504, AVERAGE_POPULARITY: 0.5248686, DIVERSITY_MEAN_INTER_LIST: 0.8784457, DIVERSITY_HERFINDAHL: 0.9878433, COVERAGE_ITEM: 0.9955997, COVERAGE_ITEM_CORRECT: 0.0541148, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4159392, DIVERSITY_GINI: 0.1498147, SHANNON_ENTROPY: 8.6560195, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 450 of 1500. Elapsed time 1.54 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 9.79E+00. Sample per second: 13904\n",
      "MF_BPR: Epoch 451 of 1500. Elapsed time 1.54 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 9.82E+00. Sample per second: 16478\n",
      "MF_BPR: Epoch 452 of 1500. Elapsed time 1.54 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 9.78E+00. Sample per second: 15592\n",
      "MF_BPR: Epoch 453 of 1500. Elapsed time 1.54 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 9.87E+00. Sample per second: 14919\n",
      "MF_BPR: Epoch 454 of 1500. Elapsed time 1.54 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 9.94E+00. Sample per second: 14349\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.07 sec. Users per second: 1730\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.19 sec. Users per second: 1733\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2321257, PRECISION: 0.0819344, PRECISION_RECALL_MIN_DEN: 0.1032607, RECALL: 0.0665870, MAP: 0.0472195, MRR: 0.1923921, NDCG: 0.0669280, F1: 0.0734677, HIT_RATE: 0.8193437, ARHR: 0.2680777, NOVELTY: 0.0097505, AVERAGE_POPULARITY: 0.5241399, DIVERSITY_MEAN_INTER_LIST: 0.8793095, DIVERSITY_HERFINDAHL: 0.9879297, COVERAGE_ITEM: 0.9955997, COVERAGE_ITEM_CORRECT: 0.0543020, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4167978, DIVERSITY_GINI: 0.1494551, SHANNON_ENTROPY: 8.6622355, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 455 of 1500. Elapsed time 1.55 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 9.94E+00. Sample per second: 15964\n",
      "MF_BPR: Epoch 456 of 1500. Elapsed time 1.55 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 9.98E+00. Sample per second: 15235\n",
      "MF_BPR: Epoch 457 of 1500. Elapsed time 1.55 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 9.96E+00. Sample per second: 14645\n",
      "MF_BPR: Epoch 458 of 1500. Elapsed time 1.56 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 9.91E+00. Sample per second: 14047\n",
      "MF_BPR: Epoch 459 of 1500. Elapsed time 1.56 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.27 seconds. BPR loss 9.95E+00. Sample per second: 13255\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.38 sec. Users per second: 1679\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 42.05 sec. Users per second: 1656\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2321465, PRECISION: 0.0819559, PRECISION_RECALL_MIN_DEN: 0.1032423, RECALL: 0.0665417, MAP: 0.0472247, MRR: 0.1922891, NDCG: 0.0669593, F1: 0.0734488, HIT_RATE: 0.8195591, ARHR: 0.2680484, NOVELTY: 0.0097510, AVERAGE_POPULARITY: 0.5233820, DIVERSITY_MEAN_INTER_LIST: 0.8801422, DIVERSITY_HERFINDAHL: 0.9880130, COVERAGE_ITEM: 0.9955060, COVERAGE_ITEM_CORRECT: 0.0545829, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4168551, DIVERSITY_GINI: 0.1490883, SHANNON_ENTROPY: 8.6681123, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 460 of 1500. Elapsed time 1.57 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 1.02E+01. Sample per second: 14974\n",
      "MF_BPR: Epoch 461 of 1500. Elapsed time 1.57 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 1.02E+01. Sample per second: 14388\n",
      "MF_BPR: Epoch 462 of 1500. Elapsed time 1.57 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.01E+01. Sample per second: 13636\n",
      "MF_BPR: Epoch 463 of 1500. Elapsed time 1.57 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 1.01E+01. Sample per second: 15796\n",
      "MF_BPR: Epoch 464 of 1500. Elapsed time 1.57 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 1.01E+01. Sample per second: 14299\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.47 sec. Users per second: 1641\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.37 sec. Users per second: 1683\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2319650, PRECISION: 0.0821010, PRECISION_RECALL_MIN_DEN: 0.1033642, RECALL: 0.0665769, MAP: 0.0472615, MRR: 0.1922719, NDCG: 0.0669999, F1: 0.0735285, HIT_RATE: 0.8210095, ARHR: 0.2681725, NOVELTY: 0.0097502, AVERAGE_POPULARITY: 0.5230072, DIVERSITY_MEAN_INTER_LIST: 0.8804501, DIVERSITY_HERFINDAHL: 0.9880437, COVERAGE_ITEM: 0.9955060, COVERAGE_ITEM_CORRECT: 0.0545829, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4168980, DIVERSITY_GINI: 0.1486457, SHANNON_ENTROPY: 8.6704335, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 465 of 1500. Elapsed time 1.59 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.52 seconds. BPR loss 1.02E+01. Sample per second: 15447\n",
      "MF_BPR: Epoch 466 of 1500. Elapsed time 1.59 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 1.01E+01. Sample per second: 14805\n",
      "MF_BPR: Epoch 467 of 1500. Elapsed time 1.59 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 1.01E+01. Sample per second: 14130\n",
      "MF_BPR: Epoch 468 of 1500. Elapsed time 1.59 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.02E+01. Sample per second: 13811\n",
      "MF_BPR: Epoch 469 of 1500. Elapsed time 1.59 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.26 seconds. BPR loss 1.03E+01. Sample per second: 16421\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.33 sec. Users per second: 1780\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.89 sec. Users per second: 1791\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2319521, PRECISION: 0.0820852, PRECISION_RECALL_MIN_DEN: 0.1033923, RECALL: 0.0666153, MAP: 0.0472795, MRR: 0.1922798, NDCG: 0.0670356, F1: 0.0735456, HIT_RATE: 0.8208516, ARHR: 0.2681757, NOVELTY: 0.0097509, AVERAGE_POPULARITY: 0.5222193, DIVERSITY_MEAN_INTER_LIST: 0.8812184, DIVERSITY_HERFINDAHL: 0.9881206, COVERAGE_ITEM: 0.9954124, COVERAGE_ITEM_CORRECT: 0.0547702, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4169123, DIVERSITY_GINI: 0.1482984, SHANNON_ENTROPY: 8.6759291, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 470 of 1500. Elapsed time 1.60 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 1.02E+01. Sample per second: 16268\n",
      "MF_BPR: Epoch 471 of 1500. Elapsed time 1.60 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 1.03E+01. Sample per second: 16044\n",
      "MF_BPR: Epoch 472 of 1500. Elapsed time 1.61 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 1.04E+01. Sample per second: 15925\n",
      "MF_BPR: Epoch 473 of 1500. Elapsed time 1.61 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.03E+01. Sample per second: 15512\n",
      "MF_BPR: Epoch 474 of 1500. Elapsed time 1.61 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 1.04E+01. Sample per second: 15290\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.04 sec. Users per second: 1765\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.28 sec. Users per second: 1773\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2318638, PRECISION: 0.0821527, PRECISION_RECALL_MIN_DEN: 0.1035433, RECALL: 0.0667607, MAP: 0.0473294, MRR: 0.1923525, NDCG: 0.0670970, F1: 0.0736612, HIT_RATE: 0.8215265, ARHR: 0.2683132, NOVELTY: 0.0097501, AVERAGE_POPULARITY: 0.5217436, DIVERSITY_MEAN_INTER_LIST: 0.8817465, DIVERSITY_HERFINDAHL: 0.9881734, COVERAGE_ITEM: 0.9950379, COVERAGE_ITEM_CORRECT: 0.0545829, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4172415, DIVERSITY_GINI: 0.1479137, SHANNON_ENTROPY: 8.6793794, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 475 of 1500. Elapsed time 1.62 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 1.04E+01. Sample per second: 13555\n",
      "MF_BPR: Epoch 476 of 1500. Elapsed time 1.62 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.04E+01. Sample per second: 15679\n",
      "MF_BPR: Epoch 477 of 1500. Elapsed time 1.62 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 1.04E+01. Sample per second: 15044\n",
      "MF_BPR: Epoch 478 of 1500. Elapsed time 1.62 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 1.05E+01. Sample per second: 14344\n",
      "MF_BPR: Epoch 479 of 1500. Elapsed time 1.63 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.05E+01. Sample per second: 13812\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.42 sec. Users per second: 1742\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.68 sec. Users per second: 1755\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2318111, PRECISION: 0.0822058, PRECISION_RECALL_MIN_DEN: 0.1036298, RECALL: 0.0668528, MAP: 0.0473470, MRR: 0.1922929, NDCG: 0.0671440, F1: 0.0737386, HIT_RATE: 0.8220579, ARHR: 0.2683711, NOVELTY: 0.0097499, AVERAGE_POPULARITY: 0.5211559, DIVERSITY_MEAN_INTER_LIST: 0.8823065, DIVERSITY_HERFINDAHL: 0.9882294, COVERAGE_ITEM: 0.9954124, COVERAGE_ITEM_CORRECT: 0.0548638, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4171556, DIVERSITY_GINI: 0.1475683, SHANNON_ENTROPY: 8.6837370, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 480 of 1500. Elapsed time 1.64 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.17 seconds. BPR loss 1.05E+01. Sample per second: 13521\n",
      "MF_BPR: Epoch 481 of 1500. Elapsed time 1.64 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.43 seconds. BPR loss 1.04E+01. Sample per second: 15770\n",
      "MF_BPR: Epoch 482 of 1500. Elapsed time 1.64 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 1.05E+01. Sample per second: 14848\n",
      "MF_BPR: Epoch 483 of 1500. Elapsed time 1.64 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 1.07E+01. Sample per second: 14281\n",
      "MF_BPR: Epoch 484 of 1500. Elapsed time 1.64 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.06E+01. Sample per second: 13696\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.36 sec. Users per second: 1779\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.65 sec. Users per second: 1802\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2320012, PRECISION: 0.0822201, PRECISION_RECALL_MIN_DEN: 0.1036478, RECALL: 0.0668439, MAP: 0.0473815, MRR: 0.1925003, NDCG: 0.0672055, F1: 0.0737390, HIT_RATE: 0.8222015, ARHR: 0.2685307, NOVELTY: 0.0097505, AVERAGE_POPULARITY: 0.5204443, DIVERSITY_MEAN_INTER_LIST: 0.8829216, DIVERSITY_HERFINDAHL: 0.9882909, COVERAGE_ITEM: 0.9947570, COVERAGE_ITEM_CORRECT: 0.0553319, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4170554, DIVERSITY_GINI: 0.1472426, SHANNON_ENTROPY: 8.6889329, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 485 of 1500. Elapsed time 1.65 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 1.07E+01. Sample per second: 14138\n",
      "MF_BPR: Epoch 486 of 1500. Elapsed time 1.66 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 1.06E+01. Sample per second: 13917\n",
      "MF_BPR: Epoch 487 of 1500. Elapsed time 1.66 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.14 seconds. BPR loss 1.07E+01. Sample per second: 16866\n",
      "MF_BPR: Epoch 488 of 1500. Elapsed time 1.66 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 1.07E+01. Sample per second: 16487\n",
      "MF_BPR: Epoch 489 of 1500. Elapsed time 1.66 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 1.08E+01. Sample per second: 15950\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.54 sec. Users per second: 1768\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.60 sec. Users per second: 1759\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2318194, PRECISION: 0.0823408, PRECISION_RECALL_MIN_DEN: 0.1038216, RECALL: 0.0669755, MAP: 0.0474599, MRR: 0.1926454, NDCG: 0.0673106, F1: 0.0738676, HIT_RATE: 0.8234078, ARHR: 0.2688401, NOVELTY: 0.0097503, AVERAGE_POPULARITY: 0.5198538, DIVERSITY_MEAN_INTER_LIST: 0.8834712, DIVERSITY_HERFINDAHL: 0.9883459, COVERAGE_ITEM: 0.9946634, COVERAGE_ITEM_CORRECT: 0.0557064, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4175992, DIVERSITY_GINI: 0.1469047, SHANNON_ENTROPY: 8.6928837, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 490 of 1500. Elapsed time 1.67 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 1.08E+01. Sample per second: 16588\n",
      "MF_BPR: Epoch 491 of 1500. Elapsed time 1.67 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.06E+01. Sample per second: 16093\n",
      "MF_BPR: Epoch 492 of 1500. Elapsed time 1.67 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.08E+01. Sample per second: 15681\n",
      "MF_BPR: Epoch 493 of 1500. Elapsed time 1.67 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 1.08E+01. Sample per second: 15398\n",
      "MF_BPR: Epoch 494 of 1500. Elapsed time 1.68 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 1.08E+01. Sample per second: 14634\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.34 sec. Users per second: 1813\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.25 sec. Users per second: 1820\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2316643, PRECISION: 0.0823178, PRECISION_RECALL_MIN_DEN: 0.1038194, RECALL: 0.0670151, MAP: 0.0474349, MRR: 0.1924704, NDCG: 0.0673169, F1: 0.0738824, HIT_RATE: 0.8231780, ARHR: 0.2686195, NOVELTY: 0.0097511, AVERAGE_POPULARITY: 0.5191827, DIVERSITY_MEAN_INTER_LIST: 0.8841085, DIVERSITY_HERFINDAHL: 0.9884096, COVERAGE_ITEM: 0.9945698, COVERAGE_ITEM_CORRECT: 0.0553319, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4172701, DIVERSITY_GINI: 0.1465629, SHANNON_ENTROPY: 8.6979398, \n",
      "\n",
      "MF_BPR: Epoch 495 of 1500. Elapsed time 1.69 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.15 seconds. BPR loss 1.08E+01. Sample per second: 16833\n",
      "MF_BPR: Epoch 496 of 1500. Elapsed time 1.69 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 1.09E+01. Sample per second: 16215\n",
      "MF_BPR: Epoch 497 of 1500. Elapsed time 1.69 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 1.09E+01. Sample per second: 15731\n",
      "MF_BPR: Epoch 498 of 1500. Elapsed time 1.69 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 1.08E+01. Sample per second: 15329\n",
      "MF_BPR: Epoch 499 of 1500. Elapsed time 1.69 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 1.09E+01. Sample per second: 14892\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.41 sec. Users per second: 1809\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.98 sec. Users per second: 1834\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2315548, PRECISION: 0.0823480, PRECISION_RECALL_MIN_DEN: 0.1038697, RECALL: 0.0670455, MAP: 0.0474679, MRR: 0.1923468, NDCG: 0.0673185, F1: 0.0739130, HIT_RATE: 0.8234796, ARHR: 0.2686140, NOVELTY: 0.0097509, AVERAGE_POPULARITY: 0.5186058, DIVERSITY_MEAN_INTER_LIST: 0.8846611, DIVERSITY_HERFINDAHL: 0.9884648, COVERAGE_ITEM: 0.9942889, COVERAGE_ITEM_CORRECT: 0.0556128, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4169839, DIVERSITY_GINI: 0.1461609, SHANNON_ENTROPY: 8.7014249, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 500 of 1500. Elapsed time 1.70 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 1.09E+01. Sample per second: 14195\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 501 of 1500. Elapsed time 1.70 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 1.10E+01. Sample per second: 13533\n",
      "MF_BPR: Epoch 502 of 1500. Elapsed time 1.71 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.10E+01. Sample per second: 16345\n",
      "MF_BPR: Epoch 503 of 1500. Elapsed time 1.71 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.43 seconds. BPR loss 1.11E+01. Sample per second: 15772\n",
      "MF_BPR: Epoch 504 of 1500. Elapsed time 1.71 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.11E+01. Sample per second: 15371\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.09 sec. Users per second: 1795\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.47 sec. Users per second: 1810\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2317456, PRECISION: 0.0823494, PRECISION_RECALL_MIN_DEN: 0.1038501, RECALL: 0.0669977, MAP: 0.0474661, MRR: 0.1924249, NDCG: 0.0673161, F1: 0.0738845, HIT_RATE: 0.8234939, ARHR: 0.2686473, NOVELTY: 0.0097511, AVERAGE_POPULARITY: 0.5180484, DIVERSITY_MEAN_INTER_LIST: 0.8852139, DIVERSITY_HERFINDAHL: 0.9885201, COVERAGE_ITEM: 0.9943825, COVERAGE_ITEM_CORRECT: 0.0558000, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4171413, DIVERSITY_GINI: 0.1459190, SHANNON_ENTROPY: 8.7058226, \n",
      "\n",
      "MF_BPR: Epoch 505 of 1500. Elapsed time 1.72 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.18 seconds. BPR loss 1.10E+01. Sample per second: 16700\n",
      "MF_BPR: Epoch 506 of 1500. Elapsed time 1.72 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 1.10E+01. Sample per second: 16057\n",
      "MF_BPR: Epoch 507 of 1500. Elapsed time 1.72 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.11E+01. Sample per second: 15434\n",
      "MF_BPR: Epoch 508 of 1500. Elapsed time 1.72 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 1.11E+01. Sample per second: 15085\n",
      "MF_BPR: Epoch 509 of 1500. Elapsed time 1.72 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 1.10E+01. Sample per second: 14670\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.28 sec. Users per second: 1816\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.11 sec. Users per second: 1827\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2316902, PRECISION: 0.0824341, PRECISION_RECALL_MIN_DEN: 0.1039531, RECALL: 0.0670757, MAP: 0.0474938, MRR: 0.1922663, NDCG: 0.0673361, F1: 0.0739661, HIT_RATE: 0.8243412, ARHR: 0.2686128, NOVELTY: 0.0097512, AVERAGE_POPULARITY: 0.5173705, DIVERSITY_MEAN_INTER_LIST: 0.8859041, DIVERSITY_HERFINDAHL: 0.9885891, COVERAGE_ITEM: 0.9941953, COVERAGE_ITEM_CORRECT: 0.0556128, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4174848, DIVERSITY_GINI: 0.1454993, SHANNON_ENTROPY: 8.7110490, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 510 of 1500. Elapsed time 1.74 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.13 seconds. BPR loss 1.12E+01. Sample per second: 13632\n",
      "MF_BPR: Epoch 511 of 1500. Elapsed time 1.74 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.12E+01. Sample per second: 16334\n",
      "MF_BPR: Epoch 512 of 1500. Elapsed time 1.74 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 1.12E+01. Sample per second: 15607\n",
      "MF_BPR: Epoch 513 of 1500. Elapsed time 1.74 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 1.13E+01. Sample per second: 15088\n",
      "MF_BPR: Epoch 514 of 1500. Elapsed time 1.74 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 1.13E+01. Sample per second: 14329\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.42 sec. Users per second: 1841\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.55 sec. Users per second: 1854\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2315461, PRECISION: 0.0825461, PRECISION_RECALL_MIN_DEN: 0.1041474, RECALL: 0.0672268, MAP: 0.0475436, MRR: 0.1923773, NDCG: 0.0674107, F1: 0.0741030, HIT_RATE: 0.8254613, ARHR: 0.2688232, NOVELTY: 0.0097511, AVERAGE_POPULARITY: 0.5166819, DIVERSITY_MEAN_INTER_LIST: 0.8864828, DIVERSITY_HERFINDAHL: 0.9886470, COVERAGE_ITEM: 0.9945698, COVERAGE_ITEM_CORRECT: 0.0558936, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4178711, DIVERSITY_GINI: 0.1452234, SHANNON_ENTROPY: 8.7159502, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 515 of 1500. Elapsed time 1.75 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 1.12E+01. Sample per second: 14999\n",
      "MF_BPR: Epoch 516 of 1500. Elapsed time 1.75 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 1.13E+01. Sample per second: 14344\n",
      "MF_BPR: Epoch 517 of 1500. Elapsed time 1.75 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.13E+01. Sample per second: 13842\n",
      "MF_BPR: Epoch 518 of 1500. Elapsed time 1.76 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 1.13E+01. Sample per second: 16481\n",
      "MF_BPR: Epoch 519 of 1500. Elapsed time 1.76 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 1.13E+01. Sample per second: 15554\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.14 sec. Users per second: 1825\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.85 sec. Users per second: 1840\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2314016, PRECISION: 0.0826625, PRECISION_RECALL_MIN_DEN: 0.1043074, RECALL: 0.0673487, MAP: 0.0476188, MRR: 0.1923992, NDCG: 0.0674977, F1: 0.0742239, HIT_RATE: 0.8266245, ARHR: 0.2690316, NOVELTY: 0.0097511, AVERAGE_POPULARITY: 0.5160433, DIVERSITY_MEAN_INTER_LIST: 0.8872288, DIVERSITY_HERFINDAHL: 0.9887216, COVERAGE_ITEM: 0.9941953, COVERAGE_ITEM_CORRECT: 0.0557064, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4178711, DIVERSITY_GINI: 0.1448374, SHANNON_ENTROPY: 8.7214644, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 520 of 1500. Elapsed time 1.77 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 1.14E+01. Sample per second: 15192\n",
      "MF_BPR: Epoch 521 of 1500. Elapsed time 1.77 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 1.14E+01. Sample per second: 14430\n",
      "MF_BPR: Epoch 522 of 1500. Elapsed time 1.77 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 1.14E+01. Sample per second: 14163\n",
      "MF_BPR: Epoch 523 of 1500. Elapsed time 1.77 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 1.13E+01. Sample per second: 13728\n",
      "MF_BPR: Epoch 524 of 1500. Elapsed time 1.77 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.15E+01. Sample per second: 16111\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.37 sec. Users per second: 1811\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.26 sec. Users per second: 1820\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2314510, PRECISION: 0.0826266, PRECISION_RECALL_MIN_DEN: 0.1042473, RECALL: 0.0672949, MAP: 0.0475945, MRR: 0.1923314, NDCG: 0.0674556, F1: 0.0741768, HIT_RATE: 0.8262655, ARHR: 0.2689551, NOVELTY: 0.0097514, AVERAGE_POPULARITY: 0.5153177, DIVERSITY_MEAN_INTER_LIST: 0.8878637, DIVERSITY_HERFINDAHL: 0.9887851, COVERAGE_ITEM: 0.9941953, COVERAGE_ITEM_CORRECT: 0.0559873, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4176422, DIVERSITY_GINI: 0.1444721, SHANNON_ENTROPY: 8.7261005, \n",
      "\n",
      "MF_BPR: Epoch 525 of 1500. Elapsed time 1.79 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.14E+01. Sample per second: 14477\n",
      "MF_BPR: Epoch 526 of 1500. Elapsed time 1.79 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 1.15E+01. Sample per second: 13999\n",
      "MF_BPR: Epoch 527 of 1500. Elapsed time 1.79 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.28 seconds. BPR loss 1.15E+01. Sample per second: 13233\n",
      "MF_BPR: Epoch 528 of 1500. Elapsed time 1.79 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 1.15E+01. Sample per second: 15725\n",
      "MF_BPR: Epoch 529 of 1500. Elapsed time 1.79 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 1.15E+01. Sample per second: 15316\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.21 sec. Users per second: 1821\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.23 sec. Users per second: 1822\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2312864, PRECISION: 0.0826596, PRECISION_RECALL_MIN_DEN: 0.1043558, RECALL: 0.0673989, MAP: 0.0476257, MRR: 0.1922550, NDCG: 0.0674733, F1: 0.0742532, HIT_RATE: 0.8265958, ARHR: 0.2688967, NOVELTY: 0.0097527, AVERAGE_POPULARITY: 0.5143733, DIVERSITY_MEAN_INTER_LIST: 0.8886728, DIVERSITY_HERFINDAHL: 0.9888660, COVERAGE_ITEM: 0.9941953, COVERAGE_ITEM_CORRECT: 0.0558936, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4176422, DIVERSITY_GINI: 0.1441659, SHANNON_ENTROPY: 8.7323992, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 530 of 1500. Elapsed time 1.80 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 1.16E+01. Sample per second: 13959\n",
      "MF_BPR: Epoch 531 of 1500. Elapsed time 1.80 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.27 seconds. BPR loss 1.15E+01. Sample per second: 16381\n",
      "MF_BPR: Epoch 532 of 1500. Elapsed time 1.80 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 1.16E+01. Sample per second: 15480\n",
      "MF_BPR: Epoch 533 of 1500. Elapsed time 1.81 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 1.15E+01. Sample per second: 14654\n",
      "MF_BPR: Epoch 534 of 1500. Elapsed time 1.81 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.17E+01. Sample per second: 13652\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.30 sec. Users per second: 1749\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.37 sec. Users per second: 1769\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2311876, PRECISION: 0.0827213, PRECISION_RECALL_MIN_DEN: 0.1044589, RECALL: 0.0675006, MAP: 0.0476492, MRR: 0.1921121, NDCG: 0.0674958, F1: 0.0743399, HIT_RATE: 0.8272133, ARHR: 0.2688462, NOVELTY: 0.0097530, AVERAGE_POPULARITY: 0.5136757, DIVERSITY_MEAN_INTER_LIST: 0.8892717, DIVERSITY_HERFINDAHL: 0.9889259, COVERAGE_ITEM: 0.9941953, COVERAGE_ITEM_CORRECT: 0.0558936, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4178139, DIVERSITY_GINI: 0.1438106, SHANNON_ENTROPY: 8.7368627, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 535 of 1500. Elapsed time 1.82 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 1.16E+01. Sample per second: 14677\n",
      "MF_BPR: Epoch 536 of 1500. Elapsed time 1.82 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 1.17E+01. Sample per second: 14282\n",
      "MF_BPR: Epoch 537 of 1500. Elapsed time 1.82 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.17E+01. Sample per second: 13848\n",
      "MF_BPR: Epoch 538 of 1500. Elapsed time 1.82 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 1.17E+01. Sample per second: 16604\n",
      "MF_BPR: Epoch 539 of 1500. Elapsed time 1.82 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 1.16E+01. Sample per second: 15838\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.36 sec. Users per second: 1779\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.91 sec. Users per second: 1790\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2311626, PRECISION: 0.0827558, PRECISION_RECALL_MIN_DEN: 0.1044702, RECALL: 0.0675231, MAP: 0.0476525, MRR: 0.1920002, NDCG: 0.0674884, F1: 0.0743674, HIT_RATE: 0.8275580, ARHR: 0.2688142, NOVELTY: 0.0097534, AVERAGE_POPULARITY: 0.5129956, DIVERSITY_MEAN_INTER_LIST: 0.8900180, DIVERSITY_HERFINDAHL: 0.9890005, COVERAGE_ITEM: 0.9941953, COVERAGE_ITEM_CORRECT: 0.0559873, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4178568, DIVERSITY_GINI: 0.1435212, SHANNON_ENTROPY: 8.7424092, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 540 of 1500. Elapsed time 1.84 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 1.18E+01. Sample per second: 15382\n",
      "MF_BPR: Epoch 541 of 1500. Elapsed time 1.84 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 1.18E+01. Sample per second: 14897\n",
      "MF_BPR: Epoch 542 of 1500. Elapsed time 1.84 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.18E+01. Sample per second: 14479\n",
      "MF_BPR: Epoch 543 of 1500. Elapsed time 1.84 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 1.18E+01. Sample per second: 13949\n",
      "MF_BPR: Epoch 544 of 1500. Elapsed time 1.84 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.19E+01. Sample per second: 16328\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.48 sec. Users per second: 1772\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.11 sec. Users per second: 1780\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2308651, PRECISION: 0.0828463, PRECISION_RECALL_MIN_DEN: 0.1046030, RECALL: 0.0676269, MAP: 0.0476884, MRR: 0.1918062, NDCG: 0.0675293, F1: 0.0744669, HIT_RATE: 0.8284627, ARHR: 0.2687171, NOVELTY: 0.0097539, AVERAGE_POPULARITY: 0.5123197, DIVERSITY_MEAN_INTER_LIST: 0.8908070, DIVERSITY_HERFINDAHL: 0.9890794, COVERAGE_ITEM: 0.9939144, COVERAGE_ITEM_CORRECT: 0.0562681, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4179141, DIVERSITY_GINI: 0.1432549, SHANNON_ENTROPY: 8.7479766, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 545 of 1500. Elapsed time 1.85 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.18E+01. Sample per second: 14409\n",
      "MF_BPR: Epoch 546 of 1500. Elapsed time 1.85 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 1.19E+01. Sample per second: 13756\n",
      "MF_BPR: Epoch 547 of 1500. Elapsed time 1.85 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 1.19E+01. Sample per second: 16047\n",
      "MF_BPR: Epoch 548 of 1500. Elapsed time 1.86 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 1.20E+01. Sample per second: 15279\n",
      "MF_BPR: Epoch 549 of 1500. Elapsed time 1.86 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.19E+01. Sample per second: 14549\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.45 sec. Users per second: 1708\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.69 sec. Users per second: 1711\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2309449, PRECISION: 0.0828635, PRECISION_RECALL_MIN_DEN: 0.1046003, RECALL: 0.0676093, MAP: 0.0477152, MRR: 0.1917851, NDCG: 0.0675278, F1: 0.0744632, HIT_RATE: 0.8286350, ARHR: 0.2687902, NOVELTY: 0.0097544, AVERAGE_POPULARITY: 0.5115998, DIVERSITY_MEAN_INTER_LIST: 0.8913146, DIVERSITY_HERFINDAHL: 0.9891302, COVERAGE_ITEM: 0.9936336, COVERAGE_ITEM_CORRECT: 0.0565490, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4177853, DIVERSITY_GINI: 0.1429047, SHANNON_ENTROPY: 8.7524398, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 550 of 1500. Elapsed time 1.87 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 1.20E+01. Sample per second: 13568\n",
      "MF_BPR: Epoch 551 of 1500. Elapsed time 1.87 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 1.20E+01. Sample per second: 15938\n",
      "MF_BPR: Epoch 552 of 1500. Elapsed time 1.87 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 1.19E+01. Sample per second: 15334\n",
      "MF_BPR: Epoch 553 of 1500. Elapsed time 1.87 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.19E+01. Sample per second: 14758\n",
      "MF_BPR: Epoch 554 of 1500. Elapsed time 1.87 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 1.20E+01. Sample per second: 14303\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.10 sec. Users per second: 1728\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.65 sec. Users per second: 1756\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2306817, PRECISION: 0.0829985, PRECISION_RECALL_MIN_DEN: 0.1047539, RECALL: 0.0676916, MAP: 0.0477048, MRR: 0.1917353, NDCG: 0.0675836, F1: 0.0745676, HIT_RATE: 0.8299849, ARHR: 0.2687943, NOVELTY: 0.0097540, AVERAGE_POPULARITY: 0.5110979, DIVERSITY_MEAN_INTER_LIST: 0.8916962, DIVERSITY_HERFINDAHL: 0.9891683, COVERAGE_ITEM: 0.9938208, COVERAGE_ITEM_CORRECT: 0.0565490, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4182289, DIVERSITY_GINI: 0.1425936, SHANNON_ENTROPY: 8.7551627, \n",
      "\n",
      "MF_BPR: Epoch 555 of 1500. Elapsed time 1.89 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 1.20E+01. Sample per second: 14797\n",
      "MF_BPR: Epoch 556 of 1500. Elapsed time 1.89 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.20E+01. Sample per second: 14473\n",
      "MF_BPR: Epoch 557 of 1500. Elapsed time 1.89 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 1.21E+01. Sample per second: 14171\n",
      "MF_BPR: Epoch 558 of 1500. Elapsed time 1.89 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 1.21E+01. Sample per second: 13853\n",
      "MF_BPR: Epoch 559 of 1500. Elapsed time 1.89 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.16 seconds. BPR loss 1.21E+01. Sample per second: 16779\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.18 sec. Users per second: 1822\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.98 sec. Users per second: 1834\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2306723, PRECISION: 0.0829813, PRECISION_RECALL_MIN_DEN: 0.1047060, RECALL: 0.0676550, MAP: 0.0476946, MRR: 0.1916771, NDCG: 0.0675601, F1: 0.0745384, HIT_RATE: 0.8298126, ARHR: 0.2687435, NOVELTY: 0.0097549, AVERAGE_POPULARITY: 0.5101353, DIVERSITY_MEAN_INTER_LIST: 0.8925510, DIVERSITY_HERFINDAHL: 0.9892538, COVERAGE_ITEM: 0.9935399, COVERAGE_ITEM_CORRECT: 0.0571108, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4181144, DIVERSITY_GINI: 0.1422646, SHANNON_ENTROPY: 8.7617303, \n",
      "\n",
      "MF_BPR: Epoch 560 of 1500. Elapsed time 1.90 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.22E+01. Sample per second: 16316\n",
      "MF_BPR: Epoch 561 of 1500. Elapsed time 1.90 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 1.22E+01. Sample per second: 15949\n",
      "MF_BPR: Epoch 562 of 1500. Elapsed time 1.90 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.22E+01. Sample per second: 15439\n",
      "MF_BPR: Epoch 563 of 1500. Elapsed time 1.91 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 1.21E+01. Sample per second: 15276\n",
      "MF_BPR: Epoch 564 of 1500. Elapsed time 1.91 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 1.22E+01. Sample per second: 14916\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.06 sec. Users per second: 1796\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.58 sec. Users per second: 1805\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2306896, PRECISION: 0.0830646, PRECISION_RECALL_MIN_DEN: 0.1048383, RECALL: 0.0677333, MAP: 0.0477477, MRR: 0.1918156, NDCG: 0.0675760, F1: 0.0746196, HIT_RATE: 0.8306455, ARHR: 0.2689558, NOVELTY: 0.0097553, AVERAGE_POPULARITY: 0.5093677, DIVERSITY_MEAN_INTER_LIST: 0.8933015, DIVERSITY_HERFINDAHL: 0.9893289, COVERAGE_ITEM: 0.9937272, COVERAGE_ITEM_CORRECT: 0.0574853, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4186296, DIVERSITY_GINI: 0.1419114, SHANNON_ENTROPY: 8.7666158, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 565 of 1500. Elapsed time 1.92 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.23E+01. Sample per second: 15546\n",
      "MF_BPR: Epoch 566 of 1500. Elapsed time 1.92 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 1.22E+01. Sample per second: 14335\n",
      "MF_BPR: Epoch 567 of 1500. Elapsed time 1.92 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 1.23E+01. Sample per second: 13725\n",
      "MF_BPR: Epoch 568 of 1500. Elapsed time 1.92 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 1.23E+01. Sample per second: 16038\n",
      "MF_BPR: Epoch 569 of 1500. Elapsed time 1.92 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 1.24E+01. Sample per second: 15238\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.47 sec. Users per second: 1838\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.73 sec. Users per second: 1846\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2307270, PRECISION: 0.0830416, PRECISION_RECALL_MIN_DEN: 0.1048023, RECALL: 0.0677117, MAP: 0.0477239, MRR: 0.1917600, NDCG: 0.0675621, F1: 0.0745972, HIT_RATE: 0.8304157, ARHR: 0.2688802, NOVELTY: 0.0097553, AVERAGE_POPULARITY: 0.5087472, DIVERSITY_MEAN_INTER_LIST: 0.8938080, DIVERSITY_HERFINDAHL: 0.9893795, COVERAGE_ITEM: 0.9938208, COVERAGE_ITEM_CORRECT: 0.0575789, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4187727, DIVERSITY_GINI: 0.1416029, SHANNON_ENTROPY: 8.7708577, \n",
      "\n",
      "MF_BPR: Epoch 570 of 1500. Elapsed time 1.94 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 1.23E+01. Sample per second: 15826\n",
      "MF_BPR: Epoch 571 of 1500. Elapsed time 1.94 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 1.23E+01. Sample per second: 15489\n",
      "MF_BPR: Epoch 572 of 1500. Elapsed time 1.94 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 1.24E+01. Sample per second: 15216\n",
      "MF_BPR: Epoch 573 of 1500. Elapsed time 1.94 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.23E+01. Sample per second: 14741\n",
      "MF_BPR: Epoch 574 of 1500. Elapsed time 1.94 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.24E+01. Sample per second: 14405\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.42 sec. Users per second: 1775\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.82 sec. Users per second: 1794\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2304433, PRECISION: 0.0831076, PRECISION_RECALL_MIN_DEN: 0.1048678, RECALL: 0.0677561, MAP: 0.0477432, MRR: 0.1917159, NDCG: 0.0675901, F1: 0.0746508, HIT_RATE: 0.8310763, ARHR: 0.2688409, NOVELTY: 0.0097550, AVERAGE_POPULARITY: 0.5082461, DIVERSITY_MEAN_INTER_LIST: 0.8943302, DIVERSITY_HERFINDAHL: 0.9894317, COVERAGE_ITEM: 0.9934463, COVERAGE_ITEM_CORRECT: 0.0577661, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4187012, DIVERSITY_GINI: 0.1413751, SHANNON_ENTROPY: 8.7748232, \n",
      "\n",
      "MF_BPR: Epoch 575 of 1500. Elapsed time 1.95 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.24E+01. Sample per second: 14544\n",
      "MF_BPR: Epoch 576 of 1500. Elapsed time 1.95 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 1.24E+01. Sample per second: 14136\n",
      "MF_BPR: Epoch 577 of 1500. Elapsed time 1.95 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.27 seconds. BPR loss 1.24E+01. Sample per second: 13258\n",
      "MF_BPR: Epoch 578 of 1500. Elapsed time 1.96 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 1.24E+01. Sample per second: 15688\n",
      "MF_BPR: Epoch 579 of 1500. Elapsed time 1.96 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.24E+01. Sample per second: 15174\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.13 sec. Users per second: 1759\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.60 sec. Users per second: 1759\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2307256, PRECISION: 0.0831148, PRECISION_RECALL_MIN_DEN: 0.1049079, RECALL: 0.0678198, MAP: 0.0477533, MRR: 0.1915620, NDCG: 0.0675870, F1: 0.0746923, HIT_RATE: 0.8311481, ARHR: 0.2687569, NOVELTY: 0.0097556, AVERAGE_POPULARITY: 0.5074314, DIVERSITY_MEAN_INTER_LIST: 0.8950867, DIVERSITY_HERFINDAHL: 0.9895074, COVERAGE_ITEM: 0.9930718, COVERAGE_ITEM_CORRECT: 0.0572980, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4186439, DIVERSITY_GINI: 0.1409890, SHANNON_ENTROPY: 8.7799785, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 580 of 1500. Elapsed time 1.97 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 1.26E+01. Sample per second: 15726\n",
      "MF_BPR: Epoch 581 of 1500. Elapsed time 1.97 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.58 seconds. BPR loss 1.26E+01. Sample per second: 15274\n",
      "MF_BPR: Epoch 582 of 1500. Elapsed time 1.97 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 1.25E+01. Sample per second: 14601\n",
      "MF_BPR: Epoch 583 of 1500. Elapsed time 1.97 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 1.25E+01. Sample per second: 14146\n",
      "MF_BPR: Epoch 584 of 1500. Elapsed time 1.97 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.25E+01. Sample per second: 13849\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.06 sec. Users per second: 1797\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.69 sec. Users per second: 1800\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2306043, PRECISION: 0.0831938, PRECISION_RECALL_MIN_DEN: 0.1050288, RECALL: 0.0678867, MAP: 0.0478079, MRR: 0.1916394, NDCG: 0.0676205, F1: 0.0747648, HIT_RATE: 0.8319380, ARHR: 0.2689437, NOVELTY: 0.0097557, AVERAGE_POPULARITY: 0.5066466, DIVERSITY_MEAN_INTER_LIST: 0.8958332, DIVERSITY_HERFINDAHL: 0.9895820, COVERAGE_ITEM: 0.9931654, COVERAGE_ITEM_CORRECT: 0.0576725, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4189015, DIVERSITY_GINI: 0.1405987, SHANNON_ENTROPY: 8.7847566, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 585 of 1500. Elapsed time 1.99 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 1.26E+01. Sample per second: 14062\n",
      "MF_BPR: Epoch 586 of 1500. Elapsed time 1.99 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 1.26E+01. Sample per second: 13557\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 587 of 1500. Elapsed time 1.99 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.26E+01. Sample per second: 16093\n",
      "MF_BPR: Epoch 588 of 1500. Elapsed time 1.99 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 1.26E+01. Sample per second: 15600\n",
      "MF_BPR: Epoch 589 of 1500. Elapsed time 1.99 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.27E+01. Sample per second: 15351\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.21 sec. Users per second: 1754\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.09 sec. Users per second: 1781\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2307708, PRECISION: 0.0831952, PRECISION_RECALL_MIN_DEN: 0.1050645, RECALL: 0.0679451, MAP: 0.0478346, MRR: 0.1917543, NDCG: 0.0676175, F1: 0.0748008, HIT_RATE: 0.8319523, ARHR: 0.2690866, NOVELTY: 0.0097551, AVERAGE_POPULARITY: 0.5062072, DIVERSITY_MEAN_INTER_LIST: 0.8962022, DIVERSITY_HERFINDAHL: 0.9896189, COVERAGE_ITEM: 0.9923228, COVERAGE_ITEM_CORRECT: 0.0575789, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4188729, DIVERSITY_GINI: 0.1401624, SHANNON_ENTROPY: 8.7871222, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 590 of 1500. Elapsed time 2.00 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 1.27E+01. Sample per second: 14368\n",
      "MF_BPR: Epoch 591 of 1500. Elapsed time 2.00 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 1.27E+01. Sample per second: 14064\n",
      "MF_BPR: Epoch 592 of 1500. Elapsed time 2.00 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.26E+01. Sample per second: 13658\n",
      "MF_BPR: Epoch 593 of 1500. Elapsed time 2.01 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 1.27E+01. Sample per second: 15603\n",
      "MF_BPR: Epoch 594 of 1500. Elapsed time 2.01 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 1.26E+01. Sample per second: 15120\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.00 sec. Users per second: 1800\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.46 sec. Users per second: 1811\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2305474, PRECISION: 0.0832512, PRECISION_RECALL_MIN_DEN: 0.1051128, RECALL: 0.0679714, MAP: 0.0478534, MRR: 0.1915151, NDCG: 0.0675667, F1: 0.0748394, HIT_RATE: 0.8325124, ARHR: 0.2689167, NOVELTY: 0.0097559, AVERAGE_POPULARITY: 0.5053423, DIVERSITY_MEAN_INTER_LIST: 0.8968812, DIVERSITY_HERFINDAHL: 0.9896868, COVERAGE_ITEM: 0.9924164, COVERAGE_ITEM_CORRECT: 0.0579534, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4186725, DIVERSITY_GINI: 0.1399655, SHANNON_ENTROPY: 8.7924116, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 595 of 1500. Elapsed time 2.02 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.26 seconds. BPR loss 1.28E+01. Sample per second: 16396\n",
      "MF_BPR: Epoch 596 of 1500. Elapsed time 2.02 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 1.28E+01. Sample per second: 16152\n",
      "MF_BPR: Epoch 597 of 1500. Elapsed time 2.02 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.28E+01. Sample per second: 15348\n",
      "MF_BPR: Epoch 598 of 1500. Elapsed time 2.02 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.27E+01. Sample per second: 14414\n",
      "MF_BPR: Epoch 599 of 1500. Elapsed time 2.02 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.96 seconds. BPR loss 1.28E+01. Sample per second: 14102\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.15 sec. Users per second: 1791\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.57 sec. Users per second: 1806\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2308476, PRECISION: 0.0831766, PRECISION_RECALL_MIN_DEN: 0.1050645, RECALL: 0.0679560, MAP: 0.0478682, MRR: 0.1915857, NDCG: 0.0675418, F1: 0.0747999, HIT_RATE: 0.8317656, ARHR: 0.2690144, NOVELTY: 0.0097558, AVERAGE_POPULARITY: 0.5048405, DIVERSITY_MEAN_INTER_LIST: 0.8973407, DIVERSITY_HERFINDAHL: 0.9897328, COVERAGE_ITEM: 0.9922292, COVERAGE_ITEM_CORRECT: 0.0581406, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4185581, DIVERSITY_GINI: 0.1396516, SHANNON_ENTROPY: 8.7959902, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 600 of 1500. Elapsed time 2.03 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 1.29E+01. Sample per second: 14645\n",
      "MF_BPR: Epoch 601 of 1500. Elapsed time 2.04 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 1.29E+01. Sample per second: 14207\n",
      "MF_BPR: Epoch 602 of 1500. Elapsed time 2.04 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 1.28E+01. Sample per second: 13767\n",
      "MF_BPR: Epoch 603 of 1500. Elapsed time 2.04 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.26 seconds. BPR loss 1.28E+01. Sample per second: 16392\n",
      "MF_BPR: Epoch 604 of 1500. Elapsed time 2.04 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 1.29E+01. Sample per second: 15750\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.38 sec. Users per second: 1778\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.11 sec. Users per second: 1780\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2306806, PRECISION: 0.0832268, PRECISION_RECALL_MIN_DEN: 0.1051182, RECALL: 0.0680087, MAP: 0.0478543, MRR: 0.1915343, NDCG: 0.0674978, F1: 0.0748521, HIT_RATE: 0.8322683, ARHR: 0.2689104, NOVELTY: 0.0097559, AVERAGE_POPULARITY: 0.5041848, DIVERSITY_MEAN_INTER_LIST: 0.8980409, DIVERSITY_HERFINDAHL: 0.9898028, COVERAGE_ITEM: 0.9922292, COVERAGE_ITEM_CORRECT: 0.0586087, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4189588, DIVERSITY_GINI: 0.1393542, SHANNON_ENTROPY: 8.8008735, \n",
      "\n",
      "MF_BPR: Epoch 605 of 1500. Elapsed time 2.05 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 1.29E+01. Sample per second: 14683\n",
      "MF_BPR: Epoch 606 of 1500. Elapsed time 2.05 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 1.30E+01. Sample per second: 14181\n",
      "MF_BPR: Epoch 607 of 1500. Elapsed time 2.05 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 1.30E+01. Sample per second: 13561\n",
      "MF_BPR: Epoch 608 of 1500. Elapsed time 2.05 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.30E+01. Sample per second: 16118\n",
      "MF_BPR: Epoch 609 of 1500. Elapsed time 2.06 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.30E+01. Sample per second: 15538\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.37 sec. Users per second: 1745\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.59 sec. Users per second: 1759\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2305961, PRECISION: 0.0833302, PRECISION_RECALL_MIN_DEN: 0.1052311, RECALL: 0.0680629, MAP: 0.0479069, MRR: 0.1914534, NDCG: 0.0675718, F1: 0.0749268, HIT_RATE: 0.8333022, ARHR: 0.2689824, NOVELTY: 0.0097564, AVERAGE_POPULARITY: 0.5034794, DIVERSITY_MEAN_INTER_LIST: 0.8986641, DIVERSITY_HERFINDAHL: 0.9898651, COVERAGE_ITEM: 0.9921356, COVERAGE_ITEM_CORRECT: 0.0587024, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4190446, DIVERSITY_GINI: 0.1391097, SHANNON_ENTROPY: 8.8065716, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 610 of 1500. Elapsed time 2.07 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 1.29E+01. Sample per second: 15633\n",
      "MF_BPR: Epoch 611 of 1500. Elapsed time 2.07 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 1.31E+01. Sample per second: 14619\n",
      "MF_BPR: Epoch 612 of 1500. Elapsed time 2.07 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 1.31E+01. Sample per second: 14005\n",
      "MF_BPR: Epoch 613 of 1500. Elapsed time 2.07 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 1.31E+01. Sample per second: 13543\n",
      "MF_BPR: Epoch 614 of 1500. Elapsed time 2.07 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.30E+01. Sample per second: 16091\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.50 sec. Users per second: 1705\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.57 sec. Users per second: 1716\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2309278, PRECISION: 0.0833518, PRECISION_RECALL_MIN_DEN: 0.1053541, RECALL: 0.0682073, MAP: 0.0479970, MRR: 0.1919000, NDCG: 0.0676358, F1: 0.0750229, HIT_RATE: 0.8335176, ARHR: 0.2695247, NOVELTY: 0.0097566, AVERAGE_POPULARITY: 0.5026725, DIVERSITY_MEAN_INTER_LIST: 0.8994531, DIVERSITY_HERFINDAHL: 0.9899440, COVERAGE_ITEM: 0.9922292, COVERAGE_ITEM_CORRECT: 0.0585151, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4192593, DIVERSITY_GINI: 0.1387860, SHANNON_ENTROPY: 8.8123271, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 615 of 1500. Elapsed time 2.09 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 1.30E+01. Sample per second: 16595\n",
      "MF_BPR: Epoch 616 of 1500. Elapsed time 2.09 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 1.32E+01. Sample per second: 15850\n",
      "MF_BPR: Epoch 617 of 1500. Elapsed time 2.09 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 1.30E+01. Sample per second: 15019\n",
      "MF_BPR: Epoch 618 of 1500. Elapsed time 2.09 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.14 seconds. BPR loss 1.31E+01. Sample per second: 13604\n",
      "MF_BPR: Epoch 619 of 1500. Elapsed time 2.09 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 1.31E+01. Sample per second: 16013\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.35 sec. Users per second: 1681\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.33 sec. Users per second: 1685\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2307708, PRECISION: 0.0833733, PRECISION_RECALL_MIN_DEN: 0.1054093, RECALL: 0.0682527, MAP: 0.0480097, MRR: 0.1918180, NDCG: 0.0676171, F1: 0.0750591, HIT_RATE: 0.8337330, ARHR: 0.2694526, NOVELTY: 0.0097573, AVERAGE_POPULARITY: 0.5019407, DIVERSITY_MEAN_INTER_LIST: 0.9001542, DIVERSITY_HERFINDAHL: 0.9900141, COVERAGE_ITEM: 0.9919483, COVERAGE_ITEM_CORRECT: 0.0588896, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4192736, DIVERSITY_GINI: 0.1384543, SHANNON_ENTROPY: 8.8174962, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 620 of 1500. Elapsed time 2.10 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 1.31E+01. Sample per second: 13944\n",
      "MF_BPR: Epoch 621 of 1500. Elapsed time 2.10 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.25 seconds. BPR loss 1.32E+01. Sample per second: 16424\n",
      "MF_BPR: Epoch 622 of 1500. Elapsed time 2.11 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 1.32E+01. Sample per second: 15744\n",
      "MF_BPR: Epoch 623 of 1500. Elapsed time 2.11 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 1.32E+01. Sample per second: 15113\n",
      "MF_BPR: Epoch 624 of 1500. Elapsed time 2.11 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.32E+01. Sample per second: 14551\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.21 sec. Users per second: 1788\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.60 sec. Users per second: 1804\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2310748, PRECISION: 0.0833877, PRECISION_RECALL_MIN_DEN: 0.1054365, RECALL: 0.0682904, MAP: 0.0480887, MRR: 0.1919863, NDCG: 0.0676842, F1: 0.0750877, HIT_RATE: 0.8338766, ARHR: 0.2697298, NOVELTY: 0.0097581, AVERAGE_POPULARITY: 0.5011724, DIVERSITY_MEAN_INTER_LIST: 0.9008932, DIVERSITY_HERFINDAHL: 0.9900880, COVERAGE_ITEM: 0.9915738, COVERAGE_ITEM_CORRECT: 0.0589832, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4191448, DIVERSITY_GINI: 0.1382209, SHANNON_ENTROPY: 8.8234192, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 625 of 1500. Elapsed time 2.12 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 1.32E+01. Sample per second: 15066\n",
      "MF_BPR: Epoch 626 of 1500. Elapsed time 2.12 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.33E+01. Sample per second: 14568\n",
      "MF_BPR: Epoch 627 of 1500. Elapsed time 2.12 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.33E+01. Sample per second: 14244\n",
      "MF_BPR: Epoch 628 of 1500. Elapsed time 2.12 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.34E+01. Sample per second: 13846\n",
      "MF_BPR: Epoch 629 of 1500. Elapsed time 2.12 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.33E+01. Sample per second: 16323\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.14 sec. Users per second: 1759\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.13 sec. Users per second: 1779\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2309228, PRECISION: 0.0834178, PRECISION_RECALL_MIN_DEN: 0.1054846, RECALL: 0.0683602, MAP: 0.0480582, MRR: 0.1917960, NDCG: 0.0676484, F1: 0.0751421, HIT_RATE: 0.8341782, ARHR: 0.2695504, NOVELTY: 0.0097577, AVERAGE_POPULARITY: 0.5006940, DIVERSITY_MEAN_INTER_LIST: 0.9012825, DIVERSITY_HERFINDAHL: 0.9901270, COVERAGE_ITEM: 0.9913866, COVERAGE_ITEM_CORRECT: 0.0586087, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4192307, DIVERSITY_GINI: 0.1378777, SHANNON_ENTROPY: 8.8262039, \n",
      "\n",
      "MF_BPR: Epoch 630 of 1500. Elapsed time 2.14 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.33E+01. Sample per second: 15143\n",
      "MF_BPR: Epoch 631 of 1500. Elapsed time 2.14 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 1.33E+01. Sample per second: 14808\n",
      "MF_BPR: Epoch 632 of 1500. Elapsed time 2.14 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 1.33E+01. Sample per second: 14491\n",
      "MF_BPR: Epoch 633 of 1500. Elapsed time 2.14 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.98 seconds. BPR loss 1.33E+01. Sample per second: 14038\n",
      "MF_BPR: Epoch 634 of 1500. Elapsed time 2.14 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.34E+01. Sample per second: 13656\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.21 sec. Users per second: 1820\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.28 sec. Users per second: 1819\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2311266, PRECISION: 0.0834078, PRECISION_RECALL_MIN_DEN: 0.1055127, RECALL: 0.0683993, MAP: 0.0481107, MRR: 0.1919851, NDCG: 0.0676708, F1: 0.0751616, HIT_RATE: 0.8340777, ARHR: 0.2698085, NOVELTY: 0.0097580, AVERAGE_POPULARITY: 0.4999398, DIVERSITY_MEAN_INTER_LIST: 0.9020060, DIVERSITY_HERFINDAHL: 0.9901993, COVERAGE_ITEM: 0.9912930, COVERAGE_ITEM_CORRECT: 0.0586087, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4191734, DIVERSITY_GINI: 0.1374790, SHANNON_ENTROPY: 8.8310953, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 635 of 1500. Elapsed time 2.15 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.34E+01. Sample per second: 15142\n",
      "MF_BPR: Epoch 636 of 1500. Elapsed time 2.15 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.34E+01. Sample per second: 14569\n",
      "MF_BPR: Epoch 637 of 1500. Elapsed time 2.15 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.34E+01. Sample per second: 14231\n",
      "MF_BPR: Epoch 638 of 1500. Elapsed time 2.16 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.34E+01. Sample per second: 13903\n",
      "MF_BPR: Epoch 639 of 1500. Elapsed time 2.16 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.16 seconds. BPR loss 1.34E+01. Sample per second: 16793\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.18 sec. Users per second: 1823\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.14 sec. Users per second: 1826\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2313461, PRECISION: 0.0833633, PRECISION_RECALL_MIN_DEN: 0.1054657, RECALL: 0.0683817, MAP: 0.0481061, MRR: 0.1919991, NDCG: 0.0676197, F1: 0.0751329, HIT_RATE: 0.8336325, ARHR: 0.2698508, NOVELTY: 0.0097581, AVERAGE_POPULARITY: 0.4994614, DIVERSITY_MEAN_INTER_LIST: 0.9024197, DIVERSITY_HERFINDAHL: 0.9902407, COVERAGE_ITEM: 0.9914802, COVERAGE_ITEM_CORRECT: 0.0588896, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4188729, DIVERSITY_GINI: 0.1372003, SHANNON_ENTROPY: 8.8342960, \n",
      "\n",
      "MF_BPR: Epoch 640 of 1500. Elapsed time 2.17 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.35E+01. Sample per second: 15657\n",
      "MF_BPR: Epoch 641 of 1500. Elapsed time 2.17 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 1.35E+01. Sample per second: 15327\n",
      "MF_BPR: Epoch 642 of 1500. Elapsed time 2.17 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 1.35E+01. Sample per second: 15043\n",
      "MF_BPR: Epoch 643 of 1500. Elapsed time 2.17 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.35E+01. Sample per second: 14728\n",
      "MF_BPR: Epoch 644 of 1500. Elapsed time 2.17 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.36E+01. Sample per second: 14245\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.29 sec. Users per second: 1816\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.15 sec. Users per second: 1825\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2314614, PRECISION: 0.0834868, PRECISION_RECALL_MIN_DEN: 0.1055677, RECALL: 0.0684229, MAP: 0.0481482, MRR: 0.1919270, NDCG: 0.0676022, F1: 0.0752079, HIT_RATE: 0.8348675, ARHR: 0.2699448, NOVELTY: 0.0097574, AVERAGE_POPULARITY: 0.4992431, DIVERSITY_MEAN_INTER_LIST: 0.9026797, DIVERSITY_HERFINDAHL: 0.9902667, COVERAGE_ITEM: 0.9911993, COVERAGE_ITEM_CORRECT: 0.0589832, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4190732, DIVERSITY_GINI: 0.1368849, SHANNON_ENTROPY: 8.8363992, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 645 of 1500. Elapsed time 2.19 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 1.36E+01. Sample per second: 16280\n",
      "MF_BPR: Epoch 646 of 1500. Elapsed time 2.19 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 1.36E+01. Sample per second: 15823\n",
      "MF_BPR: Epoch 647 of 1500. Elapsed time 2.19 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.36E+01. Sample per second: 15361\n",
      "MF_BPR: Epoch 648 of 1500. Elapsed time 2.19 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 1.36E+01. Sample per second: 14978\n",
      "MF_BPR: Epoch 649 of 1500. Elapsed time 2.19 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.36E+01. Sample per second: 14415\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.22 sec. Users per second: 1787\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.66 sec. Users per second: 1801\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2314290, PRECISION: 0.0834595, PRECISION_RECALL_MIN_DEN: 0.1055749, RECALL: 0.0684707, MAP: 0.0482018, MRR: 0.1920332, NDCG: 0.0676376, F1: 0.0752257, HIT_RATE: 0.8345947, ARHR: 0.2700923, NOVELTY: 0.0097573, AVERAGE_POPULARITY: 0.4986945, DIVERSITY_MEAN_INTER_LIST: 0.9031328, DIVERSITY_HERFINDAHL: 0.9903120, COVERAGE_ITEM: 0.9912930, COVERAGE_ITEM_CORRECT: 0.0588896, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4189731, DIVERSITY_GINI: 0.1366399, SHANNON_ENTROPY: 8.8396541, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 650 of 1500. Elapsed time 2.20 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.37E+01. Sample per second: 14745\n",
      "MF_BPR: Epoch 651 of 1500. Elapsed time 2.20 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 1.36E+01. Sample per second: 14372\n",
      "MF_BPR: Epoch 652 of 1500. Elapsed time 2.20 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 1.36E+01. Sample per second: 13958\n",
      "MF_BPR: Epoch 653 of 1500. Elapsed time 2.21 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.12 seconds. BPR loss 1.37E+01. Sample per second: 16973\n",
      "MF_BPR: Epoch 654 of 1500. Elapsed time 2.21 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 1.36E+01. Sample per second: 16491\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.14 sec. Users per second: 1758\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.12 sec. Users per second: 1780\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2313430, PRECISION: 0.0834710, PRECISION_RECALL_MIN_DEN: 0.1056025, RECALL: 0.0684903, MAP: 0.0482091, MRR: 0.1919217, NDCG: 0.0676588, F1: 0.0752422, HIT_RATE: 0.8347096, ARHR: 0.2699681, NOVELTY: 0.0097576, AVERAGE_POPULARITY: 0.4979789, DIVERSITY_MEAN_INTER_LIST: 0.9035981, DIVERSITY_HERFINDAHL: 0.9903585, COVERAGE_ITEM: 0.9915738, COVERAGE_ITEM_CORRECT: 0.0588896, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4187155, DIVERSITY_GINI: 0.1363208, SHANNON_ENTROPY: 8.8429435, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 655 of 1500. Elapsed time 2.22 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.58 seconds. BPR loss 1.37E+01. Sample per second: 15270\n",
      "MF_BPR: Epoch 656 of 1500. Elapsed time 2.22 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 1.37E+01. Sample per second: 14926\n",
      "MF_BPR: Epoch 657 of 1500. Elapsed time 2.22 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 1.38E+01. Sample per second: 14678\n",
      "MF_BPR: Epoch 658 of 1500. Elapsed time 2.22 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 1.37E+01. Sample per second: 14365\n",
      "MF_BPR: Epoch 659 of 1500. Elapsed time 2.22 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 1.38E+01. Sample per second: 13978\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.11 sec. Users per second: 1827\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.00 sec. Users per second: 1833\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2311717, PRECISION: 0.0834595, PRECISION_RECALL_MIN_DEN: 0.1055841, RECALL: 0.0684818, MAP: 0.0481636, MRR: 0.1915653, NDCG: 0.0675941, F1: 0.0752324, HIT_RATE: 0.8345947, ARHR: 0.2695376, NOVELTY: 0.0097582, AVERAGE_POPULARITY: 0.4971422, DIVERSITY_MEAN_INTER_LIST: 0.9043176, DIVERSITY_HERFINDAHL: 0.9904305, COVERAGE_ITEM: 0.9914802, COVERAGE_ITEM_CORRECT: 0.0588896, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4186582, DIVERSITY_GINI: 0.1360248, SHANNON_ENTROPY: 8.8479882, \n",
      "\n",
      "MF_BPR: Epoch 660 of 1500. Elapsed time 2.23 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.17 seconds. BPR loss 1.38E+01. Sample per second: 16744\n",
      "MF_BPR: Epoch 661 of 1500. Elapsed time 2.24 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.37E+01. Sample per second: 16325\n",
      "MF_BPR: Epoch 662 of 1500. Elapsed time 2.24 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 1.39E+01. Sample per second: 15859\n",
      "MF_BPR: Epoch 663 of 1500. Elapsed time 2.24 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 1.38E+01. Sample per second: 15563\n",
      "MF_BPR: Epoch 664 of 1500. Elapsed time 2.24 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 1.39E+01. Sample per second: 15290\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.40 sec. Users per second: 1809\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.26 sec. Users per second: 1820\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2308855, PRECISION: 0.0834954, PRECISION_RECALL_MIN_DEN: 0.1056817, RECALL: 0.0685759, MAP: 0.0482160, MRR: 0.1915960, NDCG: 0.0676406, F1: 0.0753038, HIT_RATE: 0.8349537, ARHR: 0.2696140, NOVELTY: 0.0097583, AVERAGE_POPULARITY: 0.4964728, DIVERSITY_MEAN_INTER_LIST: 0.9047594, DIVERSITY_HERFINDAHL: 0.9904746, COVERAGE_ITEM: 0.9910121, COVERAGE_ITEM_CORRECT: 0.0589832, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4186725, DIVERSITY_GINI: 0.1356895, SHANNON_ENTROPY: 8.8508721, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 665 of 1500. Elapsed time 2.25 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.39E+01. Sample per second: 13896\n",
      "MF_BPR: Epoch 666 of 1500. Elapsed time 2.25 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.38E+01. Sample per second: 16340\n",
      "MF_BPR: Epoch 667 of 1500. Elapsed time 2.25 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 1.39E+01. Sample per second: 15952\n",
      "MF_BPR: Epoch 668 of 1500. Elapsed time 2.25 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.39E+01. Sample per second: 15681\n",
      "MF_BPR: Epoch 669 of 1500. Elapsed time 2.26 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 1.39E+01. Sample per second: 15313\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.10 sec. Users per second: 1827\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.98 sec. Users per second: 1834\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2310035, PRECISION: 0.0835140, PRECISION_RECALL_MIN_DEN: 0.1056947, RECALL: 0.0685891, MAP: 0.0482414, MRR: 0.1916818, NDCG: 0.0676337, F1: 0.0753193, HIT_RATE: 0.8351404, ARHR: 0.2697233, NOVELTY: 0.0097580, AVERAGE_POPULARITY: 0.4960441, DIVERSITY_MEAN_INTER_LIST: 0.9051697, DIVERSITY_HERFINDAHL: 0.9905157, COVERAGE_ITEM: 0.9911057, COVERAGE_ITEM_CORRECT: 0.0596386, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4186582, DIVERSITY_GINI: 0.1353758, SHANNON_ENTROPY: 8.8550076, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 670 of 1500. Elapsed time 2.27 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.39E+01. Sample per second: 14747\n",
      "MF_BPR: Epoch 671 of 1500. Elapsed time 2.27 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.40E+01. Sample per second: 14408\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 672 of 1500. Elapsed time 2.27 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 1.39E+01. Sample per second: 13867\n",
      "MF_BPR: Epoch 673 of 1500. Elapsed time 2.27 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.25 seconds. BPR loss 1.40E+01. Sample per second: 16450\n",
      "MF_BPR: Epoch 674 of 1500. Elapsed time 2.27 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 1.40E+01. Sample per second: 15285\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.19 sec. Users per second: 1789\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.44 sec. Users per second: 1811\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2309557, PRECISION: 0.0835614, PRECISION_RECALL_MIN_DEN: 0.1057707, RECALL: 0.0686572, MAP: 0.0482739, MRR: 0.1916749, NDCG: 0.0676733, F1: 0.0753796, HIT_RATE: 0.8356143, ARHR: 0.2698116, NOVELTY: 0.0097585, AVERAGE_POPULARITY: 0.4952505, DIVERSITY_MEAN_INTER_LIST: 0.9059185, DIVERSITY_HERFINDAHL: 0.9905906, COVERAGE_ITEM: 0.9909185, COVERAGE_ITEM_CORRECT: 0.0596386, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4189301, DIVERSITY_GINI: 0.1350325, SHANNON_ENTROPY: 8.8599470, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 675 of 1500. Elapsed time 2.28 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.40E+01. Sample per second: 16314\n",
      "MF_BPR: Epoch 676 of 1500. Elapsed time 2.28 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 1.40E+01. Sample per second: 15894\n",
      "MF_BPR: Epoch 677 of 1500. Elapsed time 2.29 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 1.39E+01. Sample per second: 15407\n",
      "MF_BPR: Epoch 678 of 1500. Elapsed time 2.29 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 1.40E+01. Sample per second: 15008\n",
      "MF_BPR: Epoch 679 of 1500. Elapsed time 2.29 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 1.40E+01. Sample per second: 14660\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.47 sec. Users per second: 1805\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.44 sec. Users per second: 1812\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2309158, PRECISION: 0.0834968, PRECISION_RECALL_MIN_DEN: 0.1056267, RECALL: 0.0685253, MAP: 0.0482650, MRR: 0.1917954, NDCG: 0.0676027, F1: 0.0752738, HIT_RATE: 0.8349680, ARHR: 0.2698800, NOVELTY: 0.0097582, AVERAGE_POPULARITY: 0.4947930, DIVERSITY_MEAN_INTER_LIST: 0.9063222, DIVERSITY_HERFINDAHL: 0.9906309, COVERAGE_ITEM: 0.9913866, COVERAGE_ITEM_CORRECT: 0.0597322, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4184579, DIVERSITY_GINI: 0.1346155, SHANNON_ENTROPY: 8.8627339, \n",
      "\n",
      "MF_BPR: Epoch 680 of 1500. Elapsed time 2.30 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 1.41E+01. Sample per second: 15995\n",
      "MF_BPR: Epoch 681 of 1500. Elapsed time 2.30 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.41E+01. Sample per second: 15534\n",
      "MF_BPR: Epoch 682 of 1500. Elapsed time 2.30 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 1.41E+01. Sample per second: 14974\n",
      "MF_BPR: Epoch 683 of 1500. Elapsed time 2.30 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.41E+01. Sample per second: 14465\n",
      "MF_BPR: Epoch 684 of 1500. Elapsed time 2.30 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 1.42E+01. Sample per second: 13972\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.19 sec. Users per second: 1789\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.91 sec. Users per second: 1790\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2312437, PRECISION: 0.0835557, PRECISION_RECALL_MIN_DEN: 0.1057535, RECALL: 0.0686578, MAP: 0.0483190, MRR: 0.1920001, NDCG: 0.0676688, F1: 0.0753777, HIT_RATE: 0.8355568, ARHR: 0.2701733, NOVELTY: 0.0097582, AVERAGE_POPULARITY: 0.4940812, DIVERSITY_MEAN_INTER_LIST: 0.9069806, DIVERSITY_HERFINDAHL: 0.9906968, COVERAGE_ITEM: 0.9907312, COVERAGE_ITEM_CORRECT: 0.0601067, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4188300, DIVERSITY_GINI: 0.1342528, SHANNON_ENTROPY: 8.8666309, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 685 of 1500. Elapsed time 2.32 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.20 seconds. BPR loss 1.42E+01. Sample per second: 16625\n",
      "MF_BPR: Epoch 686 of 1500. Elapsed time 2.32 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 1.41E+01. Sample per second: 16188\n",
      "MF_BPR: Epoch 687 of 1500. Elapsed time 2.32 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 1.42E+01. Sample per second: 15697\n",
      "MF_BPR: Epoch 688 of 1500. Elapsed time 2.32 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.41E+01. Sample per second: 15352\n",
      "MF_BPR: Epoch 689 of 1500. Elapsed time 2.32 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 1.42E+01. Sample per second: 14948\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.22 sec. Users per second: 1787\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.76 sec. Users per second: 1796\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2312496, PRECISION: 0.0834781, PRECISION_RECALL_MIN_DEN: 0.1056538, RECALL: 0.0685789, MAP: 0.0482910, MRR: 0.1918507, NDCG: 0.0675978, F1: 0.0752986, HIT_RATE: 0.8347814, ARHR: 0.2699819, NOVELTY: 0.0097580, AVERAGE_POPULARITY: 0.4936182, DIVERSITY_MEAN_INTER_LIST: 0.9072599, DIVERSITY_HERFINDAHL: 0.9907247, COVERAGE_ITEM: 0.9909185, COVERAGE_ITEM_CORRECT: 0.0599195, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4185008, DIVERSITY_GINI: 0.1339959, SHANNON_ENTROPY: 8.8685609, \n",
      "\n",
      "MF_BPR: Epoch 690 of 1500. Elapsed time 2.33 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 1.42E+01. Sample per second: 14845\n",
      "MF_BPR: Epoch 691 of 1500. Elapsed time 2.33 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 1.43E+01. Sample per second: 14514\n",
      "MF_BPR: Epoch 692 of 1500. Elapsed time 2.34 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 1.42E+01. Sample per second: 14069\n",
      "MF_BPR: Epoch 693 of 1500. Elapsed time 2.34 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.43E+01. Sample per second: 13712\n",
      "MF_BPR: Epoch 694 of 1500. Elapsed time 2.34 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.17 seconds. BPR loss 1.42E+01. Sample per second: 16739\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.26 sec. Users per second: 1751\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.38 sec. Users per second: 1768\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2313777, PRECISION: 0.0835270, PRECISION_RECALL_MIN_DEN: 0.1057145, RECALL: 0.0686460, MAP: 0.0483612, MRR: 0.1920713, NDCG: 0.0676396, F1: 0.0753589, HIT_RATE: 0.8352696, ARHR: 0.2702867, NOVELTY: 0.0097577, AVERAGE_POPULARITY: 0.4931680, DIVERSITY_MEAN_INTER_LIST: 0.9075600, DIVERSITY_HERFINDAHL: 0.9907547, COVERAGE_ITEM: 0.9909185, COVERAGE_ITEM_CORRECT: 0.0597322, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4186153, DIVERSITY_GINI: 0.1336986, SHANNON_ENTROPY: 8.8702237, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 695 of 1500. Elapsed time 2.35 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 1.43E+01. Sample per second: 14604\n",
      "MF_BPR: Epoch 696 of 1500. Elapsed time 2.35 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 1.43E+01. Sample per second: 14364\n",
      "MF_BPR: Epoch 697 of 1500. Elapsed time 2.35 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 1.44E+01. Sample per second: 13983\n",
      "MF_BPR: Epoch 698 of 1500. Elapsed time 2.35 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.16 seconds. BPR loss 1.43E+01. Sample per second: 16791\n",
      "MF_BPR: Epoch 699 of 1500. Elapsed time 2.35 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 1.43E+01. Sample per second: 16213\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.30 sec. Users per second: 1815\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.91 sec. Users per second: 1790\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2313330, PRECISION: 0.0835227, PRECISION_RECALL_MIN_DEN: 0.1056926, RECALL: 0.0686048, MAP: 0.0483933, MRR: 0.1919388, NDCG: 0.0675873, F1: 0.0753323, HIT_RATE: 0.8352265, ARHR: 0.2702608, NOVELTY: 0.0097580, AVERAGE_POPULARITY: 0.4925197, DIVERSITY_MEAN_INTER_LIST: 0.9081609, DIVERSITY_HERFINDAHL: 0.9908148, COVERAGE_ITEM: 0.9903567, COVERAGE_ITEM_CORRECT: 0.0599195, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4183291, DIVERSITY_GINI: 0.1334257, SHANNON_ENTROPY: 8.8742132, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 700 of 1500. Elapsed time 2.37 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 1.45E+01. Sample per second: 15560\n",
      "MF_BPR: Epoch 701 of 1500. Elapsed time 2.37 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 1.44E+01. Sample per second: 14989\n",
      "MF_BPR: Epoch 702 of 1500. Elapsed time 2.37 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.45E+01. Sample per second: 14409\n",
      "MF_BPR: Epoch 703 of 1500. Elapsed time 2.37 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 1.44E+01. Sample per second: 13788\n",
      "MF_BPR: Epoch 704 of 1500. Elapsed time 2.37 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.45E+01. Sample per second: 16332\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.24 sec. Users per second: 1819\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.06 sec. Users per second: 1829\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2313032, PRECISION: 0.0835069, PRECISION_RECALL_MIN_DEN: 0.1056942, RECALL: 0.0686465, MAP: 0.0484124, MRR: 0.1919683, NDCG: 0.0675614, F1: 0.0753510, HIT_RATE: 0.8350686, ARHR: 0.2703154, NOVELTY: 0.0097576, AVERAGE_POPULARITY: 0.4922203, DIVERSITY_MEAN_INTER_LIST: 0.9084577, DIVERSITY_HERFINDAHL: 0.9908445, COVERAGE_ITEM: 0.9907312, COVERAGE_ITEM_CORRECT: 0.0599195, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4181144, DIVERSITY_GINI: 0.1331139, SHANNON_ENTROPY: 8.8766716, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 705 of 1500. Elapsed time 2.38 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.45E+01. Sample per second: 15430\n",
      "MF_BPR: Epoch 706 of 1500. Elapsed time 2.38 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.58 seconds. BPR loss 1.45E+01. Sample per second: 15259\n",
      "MF_BPR: Epoch 707 of 1500. Elapsed time 2.39 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 1.46E+01. Sample per second: 14914\n",
      "MF_BPR: Epoch 708 of 1500. Elapsed time 2.39 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.45E+01. Sample per second: 14469\n",
      "MF_BPR: Epoch 709 of 1500. Elapsed time 2.39 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.46E+01. Sample per second: 14250\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.26 sec. Users per second: 1818\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.35 sec. Users per second: 1816\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2311908, PRECISION: 0.0836490, PRECISION_RECALL_MIN_DEN: 0.1058579, RECALL: 0.0687489, MAP: 0.0484708, MRR: 0.1921673, NDCG: 0.0676595, F1: 0.0754705, HIT_RATE: 0.8364903, ARHR: 0.2706395, NOVELTY: 0.0097576, AVERAGE_POPULARITY: 0.4915974, DIVERSITY_MEAN_INTER_LIST: 0.9089766, DIVERSITY_HERFINDAHL: 0.9908964, COVERAGE_ITEM: 0.9906376, COVERAGE_ITEM_CORRECT: 0.0603876, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4185437, DIVERSITY_GINI: 0.1328441, SHANNON_ENTROPY: 8.8813888, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 710 of 1500. Elapsed time 2.40 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 1.45E+01. Sample per second: 15503\n",
      "MF_BPR: Epoch 711 of 1500. Elapsed time 2.40 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 1.45E+01. Sample per second: 14963\n",
      "MF_BPR: Epoch 712 of 1500. Elapsed time 2.40 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.45E+01. Sample per second: 14567\n",
      "MF_BPR: Epoch 713 of 1500. Elapsed time 2.40 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.46E+01. Sample per second: 14247\n",
      "MF_BPR: Epoch 714 of 1500. Elapsed time 2.40 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 1.45E+01. Sample per second: 13788\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.32 sec. Users per second: 1781\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.70 sec. Users per second: 1799\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2310814, PRECISION: 0.0836462, PRECISION_RECALL_MIN_DEN: 0.1058680, RECALL: 0.0687459, MAP: 0.0484461, MRR: 0.1920395, NDCG: 0.0676420, F1: 0.0754676, HIT_RATE: 0.8364615, ARHR: 0.2705070, NOVELTY: 0.0097567, AVERAGE_POPULARITY: 0.4911738, DIVERSITY_MEAN_INTER_LIST: 0.9093158, DIVERSITY_HERFINDAHL: 0.9909303, COVERAGE_ITEM: 0.9902631, COVERAGE_ITEM_CORRECT: 0.0604812, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4184006, DIVERSITY_GINI: 0.1324969, SHANNON_ENTROPY: 8.8833261, \n",
      "\n",
      "MF_BPR: Epoch 715 of 1500. Elapsed time 2.42 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.46E+01. Sample per second: 14243\n",
      "MF_BPR: Epoch 716 of 1500. Elapsed time 2.42 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.46E+01. Sample per second: 13806\n",
      "MF_BPR: Epoch 717 of 1500. Elapsed time 2.42 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.16 seconds. BPR loss 1.46E+01. Sample per second: 16789\n",
      "MF_BPR: Epoch 718 of 1500. Elapsed time 2.42 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 1.47E+01. Sample per second: 16224\n",
      "MF_BPR: Epoch 719 of 1500. Elapsed time 2.42 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 1.47E+01. Sample per second: 15731\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.45 sec. Users per second: 1806\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.39 sec. Users per second: 1814\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2310667, PRECISION: 0.0837122, PRECISION_RECALL_MIN_DEN: 0.1059865, RECALL: 0.0688418, MAP: 0.0485020, MRR: 0.1921363, NDCG: 0.0677354, F1: 0.0755522, HIT_RATE: 0.8371221, ARHR: 0.2706676, NOVELTY: 0.0097561, AVERAGE_POPULARITY: 0.4907990, DIVERSITY_MEAN_INTER_LIST: 0.9096208, DIVERSITY_HERFINDAHL: 0.9909608, COVERAGE_ITEM: 0.9901695, COVERAGE_ITEM_CORRECT: 0.0603876, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4187727, DIVERSITY_GINI: 0.1321882, SHANNON_ENTROPY: 8.8842631, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 720 of 1500. Elapsed time 2.43 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.34 seconds. BPR loss 1.47E+01. Sample per second: 13088\n",
      "MF_BPR: Epoch 721 of 1500. Elapsed time 2.43 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.52 seconds. BPR loss 1.48E+01. Sample per second: 15467\n",
      "MF_BPR: Epoch 722 of 1500. Elapsed time 2.43 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 1.47E+01. Sample per second: 14839\n",
      "MF_BPR: Epoch 723 of 1500. Elapsed time 2.44 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.43 seconds. BPR loss 1.47E+01. Sample per second: 12862\n",
      "MF_BPR: Epoch 724 of 1500. Elapsed time 2.44 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 1.47E+01. Sample per second: 14963\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 50000 ( 71.80% ) in 30.30 sec. Users per second: 1650\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 41.38 sec. Users per second: 1683\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2313194, PRECISION: 0.0836921, PRECISION_RECALL_MIN_DEN: 0.1060070, RECALL: 0.0688890, MAP: 0.0485032, MRR: 0.1921048, NDCG: 0.0676769, F1: 0.0755725, HIT_RATE: 0.8369211, ARHR: 0.2706518, NOVELTY: 0.0097568, AVERAGE_POPULARITY: 0.4900370, DIVERSITY_MEAN_INTER_LIST: 0.9102435, DIVERSITY_HERFINDAHL: 0.9910230, COVERAGE_ITEM: 0.9900758, COVERAGE_ITEM_CORRECT: 0.0608557, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4186869, DIVERSITY_GINI: 0.1318951, SHANNON_ENTROPY: 8.8897774, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 725 of 1500. Elapsed time 2.45 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 1.48E+01. Sample per second: 15949\n",
      "MF_BPR: Epoch 726 of 1500. Elapsed time 2.45 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.48E+01. Sample per second: 15357\n",
      "MF_BPR: Epoch 727 of 1500. Elapsed time 2.45 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 1.48E+01. Sample per second: 13538\n",
      "MF_BPR: Epoch 728 of 1500. Elapsed time 2.45 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 1.48E+01. Sample per second: 16067\n",
      "MF_BPR: Epoch 729 of 1500. Elapsed time 2.45 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 1.48E+01. Sample per second: 15200\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.51 sec. Users per second: 1803\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.41 sec. Users per second: 1813\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2312276, PRECISION: 0.0837338, PRECISION_RECALL_MIN_DEN: 0.1060527, RECALL: 0.0689177, MAP: 0.0485157, MRR: 0.1921567, NDCG: 0.0677075, F1: 0.0756067, HIT_RATE: 0.8373375, ARHR: 0.2706738, NOVELTY: 0.0097569, AVERAGE_POPULARITY: 0.4894267, DIVERSITY_MEAN_INTER_LIST: 0.9106245, DIVERSITY_HERFINDAHL: 0.9910611, COVERAGE_ITEM: 0.9899822, COVERAGE_ITEM_CORRECT: 0.0615111, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4190303, DIVERSITY_GINI: 0.1316542, SHANNON_ENTROPY: 8.8922306, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 730 of 1500. Elapsed time 2.47 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 1.49E+01. Sample per second: 16476\n",
      "MF_BPR: Epoch 731 of 1500. Elapsed time 2.47 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.43 seconds. BPR loss 1.48E+01. Sample per second: 15783\n",
      "MF_BPR: Epoch 732 of 1500. Elapsed time 2.47 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 1.48E+01. Sample per second: 15310\n",
      "MF_BPR: Epoch 733 of 1500. Elapsed time 2.47 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 1.49E+01. Sample per second: 14711\n",
      "MF_BPR: Epoch 734 of 1500. Elapsed time 2.47 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 1.49E+01. Sample per second: 13975\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.03 sec. Users per second: 1798\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.59 sec. Users per second: 1804\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2311308, PRECISION: 0.0837208, PRECISION_RECALL_MIN_DEN: 0.1060422, RECALL: 0.0689073, MAP: 0.0485032, MRR: 0.1922677, NDCG: 0.0676891, F1: 0.0755952, HIT_RATE: 0.8372083, ARHR: 0.2707205, NOVELTY: 0.0097566, AVERAGE_POPULARITY: 0.4889668, DIVERSITY_MEAN_INTER_LIST: 0.9110723, DIVERSITY_HERFINDAHL: 0.9911059, COVERAGE_ITEM: 0.9897950, COVERAGE_ITEM_CORRECT: 0.0613238, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4188443, DIVERSITY_GINI: 0.1313897, SHANNON_ENTROPY: 8.8962775, \n",
      "\n",
      "MF_BPR: Epoch 735 of 1500. Elapsed time 2.48 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.48E+01. Sample per second: 14571\n",
      "MF_BPR: Epoch 736 of 1500. Elapsed time 2.48 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 1.49E+01. Sample per second: 14170\n",
      "MF_BPR: Epoch 737 of 1500. Elapsed time 2.49 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.49E+01. Sample per second: 13653\n",
      "MF_BPR: Epoch 738 of 1500. Elapsed time 2.49 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.50E+01. Sample per second: 16322\n",
      "MF_BPR: Epoch 739 of 1500. Elapsed time 2.49 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 1.50E+01. Sample per second: 15890\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.06 sec. Users per second: 1797\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.67 sec. Users per second: 1801\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2312351, PRECISION: 0.0837984, PRECISION_RECALL_MIN_DEN: 0.1060984, RECALL: 0.0689309, MAP: 0.0485527, MRR: 0.1923668, NDCG: 0.0677147, F1: 0.0756410, HIT_RATE: 0.8379838, ARHR: 0.2709633, NOVELTY: 0.0097557, AVERAGE_POPULARITY: 0.4887531, DIVERSITY_MEAN_INTER_LIST: 0.9113731, DIVERSITY_HERFINDAHL: 0.9911360, COVERAGE_ITEM: 0.9897013, COVERAGE_ITEM_CORRECT: 0.0613238, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4189874, DIVERSITY_GINI: 0.1310555, SHANNON_ENTROPY: 8.8982740, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 740 of 1500. Elapsed time 2.50 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 1.50E+01. Sample per second: 15958\n",
      "MF_BPR: Epoch 741 of 1500. Elapsed time 2.50 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 1.50E+01. Sample per second: 15221\n",
      "MF_BPR: Epoch 742 of 1500. Elapsed time 2.50 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.50E+01. Sample per second: 14873\n",
      "MF_BPR: Epoch 743 of 1500. Elapsed time 2.50 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 1.50E+01. Sample per second: 14524\n",
      "MF_BPR: Epoch 744 of 1500. Elapsed time 2.50 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 1.51E+01. Sample per second: 14207\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.12 sec. Users per second: 1793\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.61 sec. Users per second: 1758\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2310206, PRECISION: 0.0838041, PRECISION_RECALL_MIN_DEN: 0.1061403, RECALL: 0.0689742, MAP: 0.0485839, MRR: 0.1925779, NDCG: 0.0677226, F1: 0.0756694, HIT_RATE: 0.8380412, ARHR: 0.2711703, NOVELTY: 0.0097550, AVERAGE_POPULARITY: 0.4883162, DIVERSITY_MEAN_INTER_LIST: 0.9117417, DIVERSITY_HERFINDAHL: 0.9911729, COVERAGE_ITEM: 0.9897013, COVERAGE_ITEM_CORRECT: 0.0614175, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4187870, DIVERSITY_GINI: 0.1307775, SHANNON_ENTROPY: 8.9008815, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 745 of 1500. Elapsed time 2.52 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 1.50E+01. Sample per second: 14519\n",
      "MF_BPR: Epoch 746 of 1500. Elapsed time 2.52 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 1.50E+01. Sample per second: 13995\n",
      "MF_BPR: Epoch 747 of 1500. Elapsed time 2.52 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.19 seconds. BPR loss 1.51E+01. Sample per second: 13458\n",
      "MF_BPR: Epoch 748 of 1500. Elapsed time 2.52 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 1.51E+01. Sample per second: 16149\n",
      "MF_BPR: Epoch 749 of 1500. Elapsed time 2.52 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 1.51E+01. Sample per second: 15855\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.05 sec. Users per second: 1764\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.99 sec. Users per second: 1786\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2314255, PRECISION: 0.0837582, PRECISION_RECALL_MIN_DEN: 0.1060999, RECALL: 0.0689591, MAP: 0.0485580, MRR: 0.1925307, NDCG: 0.0676957, F1: 0.0756416, HIT_RATE: 0.8375817, ARHR: 0.2710413, NOVELTY: 0.0097549, AVERAGE_POPULARITY: 0.4880523, DIVERSITY_MEAN_INTER_LIST: 0.9118630, DIVERSITY_HERFINDAHL: 0.9911850, COVERAGE_ITEM: 0.9890460, COVERAGE_ITEM_CORRECT: 0.0612302, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4189588, DIVERSITY_GINI: 0.1304751, SHANNON_ENTROPY: 8.9013969, \n",
      "\n",
      "MF_BPR: Epoch 750 of 1500. Elapsed time 2.53 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 1.51E+01. Sample per second: 15177\n",
      "MF_BPR: Epoch 751 of 1500. Elapsed time 2.53 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 1.51E+01. Sample per second: 14719\n",
      "MF_BPR: Epoch 752 of 1500. Elapsed time 2.54 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 1.51E+01. Sample per second: 14348\n",
      "MF_BPR: Epoch 753 of 1500. Elapsed time 2.54 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.51E+01. Sample per second: 13905\n",
      "MF_BPR: Epoch 754 of 1500. Elapsed time 2.54 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 1.52E+01. Sample per second: 16615\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.42 sec. Users per second: 1808\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.29 sec. Users per second: 1819\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2313309, PRECISION: 0.0837539, PRECISION_RECALL_MIN_DEN: 0.1060752, RECALL: 0.0689528, MAP: 0.0485563, MRR: 0.1924409, NDCG: 0.0676663, F1: 0.0756360, HIT_RATE: 0.8375386, ARHR: 0.2709741, NOVELTY: 0.0097549, AVERAGE_POPULARITY: 0.4873501, DIVERSITY_MEAN_INTER_LIST: 0.9123886, DIVERSITY_HERFINDAHL: 0.9912375, COVERAGE_ITEM: 0.9886715, COVERAGE_ITEM_CORRECT: 0.0613238, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4189015, DIVERSITY_GINI: 0.1301036, SHANNON_ENTROPY: 8.9049167, \n",
      "\n",
      "MF_BPR: Epoch 755 of 1500. Elapsed time 2.55 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 1.52E+01. Sample per second: 14725\n",
      "MF_BPR: Epoch 756 of 1500. Elapsed time 2.55 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.52E+01. Sample per second: 14402\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 757 of 1500. Elapsed time 2.55 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 1.52E+01. Sample per second: 13990\n",
      "MF_BPR: Epoch 758 of 1500. Elapsed time 2.55 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 1.52E+01. Sample per second: 13559\n",
      "MF_BPR: Epoch 759 of 1500. Elapsed time 2.55 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 1.53E+01. Sample per second: 16198\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.29 sec. Users per second: 1783\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.71 sec. Users per second: 1754\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2314992, PRECISION: 0.0837668, PRECISION_RECALL_MIN_DEN: 0.1061159, RECALL: 0.0689981, MAP: 0.0486103, MRR: 0.1926762, NDCG: 0.0676989, F1: 0.0756685, HIT_RATE: 0.8376678, ARHR: 0.2712950, NOVELTY: 0.0097551, AVERAGE_POPULARITY: 0.4867230, DIVERSITY_MEAN_INTER_LIST: 0.9128493, DIVERSITY_HERFINDAHL: 0.9912836, COVERAGE_ITEM: 0.9882034, COVERAGE_ITEM_CORRECT: 0.0616047, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4189731, DIVERSITY_GINI: 0.1298222, SHANNON_ENTROPY: 8.9081786, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 760 of 1500. Elapsed time 2.57 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.53E+01. Sample per second: 16100\n",
      "MF_BPR: Epoch 761 of 1500. Elapsed time 2.57 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.52E+01. Sample per second: 15362\n",
      "MF_BPR: Epoch 762 of 1500. Elapsed time 2.57 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.52E+01. Sample per second: 14756\n",
      "MF_BPR: Epoch 763 of 1500. Elapsed time 2.57 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 1.53E+01. Sample per second: 14282\n",
      "MF_BPR: Epoch 764 of 1500. Elapsed time 2.57 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 1.53E+01. Sample per second: 13772\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.07 sec. Users per second: 1729\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.18 sec. Users per second: 1733\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2313481, PRECISION: 0.0837266, PRECISION_RECALL_MIN_DEN: 0.1060283, RECALL: 0.0689226, MAP: 0.0485829, MRR: 0.1924257, NDCG: 0.0675965, F1: 0.0756067, HIT_RATE: 0.8372657, ARHR: 0.2710525, NOVELTY: 0.0097550, AVERAGE_POPULARITY: 0.4862715, DIVERSITY_MEAN_INTER_LIST: 0.9132523, DIVERSITY_HERFINDAHL: 0.9913239, COVERAGE_ITEM: 0.9881097, COVERAGE_ITEM_CORRECT: 0.0616983, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4184579, DIVERSITY_GINI: 0.1295744, SHANNON_ENTROPY: 8.9105718, \n",
      "\n",
      "MF_BPR: Epoch 765 of 1500. Elapsed time 2.58 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.52 seconds. BPR loss 1.52E+01. Sample per second: 15458\n",
      "MF_BPR: Epoch 766 of 1500. Elapsed time 2.58 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.54E+01. Sample per second: 14866\n",
      "MF_BPR: Epoch 767 of 1500. Elapsed time 2.59 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.53E+01. Sample per second: 14475\n",
      "MF_BPR: Epoch 768 of 1500. Elapsed time 2.59 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.98 seconds. BPR loss 1.54E+01. Sample per second: 14025\n",
      "MF_BPR: Epoch 769 of 1500. Elapsed time 2.59 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.54E+01. Sample per second: 13702\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.32 sec. Users per second: 1748\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.94 sec. Users per second: 1743\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2314831, PRECISION: 0.0838185, PRECISION_RECALL_MIN_DEN: 0.1062370, RECALL: 0.0691205, MAP: 0.0486648, MRR: 0.1928429, NDCG: 0.0677386, F1: 0.0757632, HIT_RATE: 0.8381848, ARHR: 0.2715069, NOVELTY: 0.0097553, AVERAGE_POPULARITY: 0.4855711, DIVERSITY_MEAN_INTER_LIST: 0.9136595, DIVERSITY_HERFINDAHL: 0.9913646, COVERAGE_ITEM: 0.9882034, COVERAGE_ITEM_CORRECT: 0.0617920, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4187012, DIVERSITY_GINI: 0.1292953, SHANNON_ENTROPY: 8.9132085, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 770 of 1500. Elapsed time 2.60 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 1.54E+01. Sample per second: 16060\n",
      "MF_BPR: Epoch 771 of 1500. Elapsed time 2.60 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.54E+01. Sample per second: 15370\n",
      "MF_BPR: Epoch 772 of 1500. Elapsed time 2.60 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 1.54E+01. Sample per second: 14909\n",
      "MF_BPR: Epoch 773 of 1500. Elapsed time 2.60 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.53E+01. Sample per second: 14473\n",
      "MF_BPR: Epoch 774 of 1500. Elapsed time 2.60 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 1.55E+01. Sample per second: 14103\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.38 sec. Users per second: 1810\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.39 sec. Users per second: 1814\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2313235, PRECISION: 0.0837740, PRECISION_RECALL_MIN_DEN: 0.1061916, RECALL: 0.0691230, MAP: 0.0486809, MRR: 0.1929604, NDCG: 0.0678021, F1: 0.0757465, HIT_RATE: 0.8377396, ARHR: 0.2716231, NOVELTY: 0.0097555, AVERAGE_POPULARITY: 0.4849159, DIVERSITY_MEAN_INTER_LIST: 0.9141974, DIVERSITY_HERFINDAHL: 0.9914184, COVERAGE_ITEM: 0.9881097, COVERAGE_ITEM_CORRECT: 0.0619792, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4183863, DIVERSITY_GINI: 0.1289362, SHANNON_ENTROPY: 8.9171678, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 775 of 1500. Elapsed time 2.62 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 1.55E+01. Sample per second: 15072\n",
      "MF_BPR: Epoch 776 of 1500. Elapsed time 2.62 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.54E+01. Sample per second: 13710\n",
      "MF_BPR: Epoch 777 of 1500. Elapsed time 2.62 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 1.55E+01. Sample per second: 16050\n",
      "MF_BPR: Epoch 778 of 1500. Elapsed time 2.62 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.55E+01. Sample per second: 15436\n",
      "MF_BPR: Epoch 779 of 1500. Elapsed time 2.62 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.55E+01. Sample per second: 14871\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 47000 ( 67.49% ) in 30.47 sec. Users per second: 1543\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 45.18 sec. Users per second: 1541\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2314007, PRECISION: 0.0838185, PRECISION_RECALL_MIN_DEN: 0.1062539, RECALL: 0.0691624, MAP: 0.0487561, MRR: 0.1931379, NDCG: 0.0677816, F1: 0.0757884, HIT_RATE: 0.8381848, ARHR: 0.2719326, NOVELTY: 0.0097556, AVERAGE_POPULARITY: 0.4843984, DIVERSITY_MEAN_INTER_LIST: 0.9145761, DIVERSITY_HERFINDAHL: 0.9914563, COVERAGE_ITEM: 0.9876416, COVERAGE_ITEM_CORRECT: 0.0620728, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4182718, DIVERSITY_GINI: 0.1286894, SHANNON_ENTROPY: 8.9195715, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 780 of 1500. Elapsed time 2.64 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.41 seconds. BPR loss 1.55E+01. Sample per second: 12921\n",
      "MF_BPR: Epoch 781 of 1500. Elapsed time 2.64 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.56E+01. Sample per second: 14752\n",
      "MF_BPR: Epoch 782 of 1500. Elapsed time 2.64 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.47 seconds. BPR loss 1.55E+01. Sample per second: 12784\n",
      "MF_BPR: Epoch 783 of 1500. Elapsed time 2.64 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.56E+01. Sample per second: 13659\n",
      "MF_BPR: Epoch 784 of 1500. Elapsed time 2.64 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 1.56E+01. Sample per second: 15618\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.19 sec. Users per second: 1690\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.91 sec. Users per second: 1702\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2314231, PRECISION: 0.0839090, PRECISION_RECALL_MIN_DEN: 0.1062895, RECALL: 0.0691257, MAP: 0.0487325, MRR: 0.1929111, NDCG: 0.0677677, F1: 0.0758033, HIT_RATE: 0.8390895, ARHR: 0.2717689, NOVELTY: 0.0097544, AVERAGE_POPULARITY: 0.4843369, DIVERSITY_MEAN_INTER_LIST: 0.9145676, DIVERSITY_HERFINDAHL: 0.9914555, COVERAGE_ITEM: 0.9879225, COVERAGE_ITEM_CORRECT: 0.0618856, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4186725, DIVERSITY_GINI: 0.1284272, SHANNON_ENTROPY: 8.9196678, \n",
      "\n",
      "MF_BPR: Epoch 785 of 1500. Elapsed time 2.65 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 1.57E+01. Sample per second: 15076\n",
      "MF_BPR: Epoch 786 of 1500. Elapsed time 2.65 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 1.56E+01. Sample per second: 14303\n",
      "MF_BPR: Epoch 787 of 1500. Elapsed time 2.66 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 1.57E+01. Sample per second: 13535\n",
      "MF_BPR: Epoch 788 of 1500. Elapsed time 2.66 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 1.56E+01. Sample per second: 15567\n",
      "MF_BPR: Epoch 789 of 1500. Elapsed time 2.66 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 1.58E+01. Sample per second: 14840\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.06 sec. Users per second: 1763\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.15 sec. Users per second: 1779\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2314467, PRECISION: 0.0839908, PRECISION_RECALL_MIN_DEN: 0.1064823, RECALL: 0.0692889, MAP: 0.0488040, MRR: 0.1930656, NDCG: 0.0678394, F1: 0.0759348, HIT_RATE: 0.8399081, ARHR: 0.2719973, NOVELTY: 0.0097540, AVERAGE_POPULARITY: 0.4840403, DIVERSITY_MEAN_INTER_LIST: 0.9147693, DIVERSITY_HERFINDAHL: 0.9914756, COVERAGE_ITEM: 0.9877352, COVERAGE_ITEM_CORRECT: 0.0616983, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4192307, DIVERSITY_GINI: 0.1281369, SHANNON_ENTROPY: 8.9207634, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 790 of 1500. Elapsed time 2.67 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.20 seconds. BPR loss 1.57E+01. Sample per second: 13438\n",
      "MF_BPR: Epoch 791 of 1500. Elapsed time 2.67 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 1.58E+01. Sample per second: 16160\n",
      "MF_BPR: Epoch 792 of 1500. Elapsed time 2.67 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.57E+01. Sample per second: 15368\n",
      "MF_BPR: Epoch 793 of 1500. Elapsed time 2.67 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.57E+01. Sample per second: 14555\n",
      "MF_BPR: Epoch 794 of 1500. Elapsed time 2.67 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.57E+01. Sample per second: 13846\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 56000 ( 80.42% ) in 30.30 sec. Users per second: 1848\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.64 sec. Users per second: 1850\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2313894, PRECISION: 0.0839549, PRECISION_RECALL_MIN_DEN: 0.1063166, RECALL: 0.0691128, MAP: 0.0487704, MRR: 0.1928615, NDCG: 0.0676854, F1: 0.0758143, HIT_RATE: 0.8395491, ARHR: 0.2718189, NOVELTY: 0.0097532, AVERAGE_POPULARITY: 0.4836812, DIVERSITY_MEAN_INTER_LIST: 0.9152020, DIVERSITY_HERFINDAHL: 0.9915189, COVERAGE_ITEM: 0.9874544, COVERAGE_ITEM_CORRECT: 0.0617920, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4186153, DIVERSITY_GINI: 0.1279115, SHANNON_ENTROPY: 8.9232191, \n",
      "\n",
      "MF_BPR: Epoch 795 of 1500. Elapsed time 2.69 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.58E+01. Sample per second: 14245\n",
      "MF_BPR: Epoch 796 of 1500. Elapsed time 2.69 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.21 seconds. BPR loss 1.57E+01. Sample per second: 13405\n",
      "MF_BPR: Epoch 797 of 1500. Elapsed time 2.69 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 1.58E+01. Sample per second: 15484\n",
      "MF_BPR: Epoch 798 of 1500. Elapsed time 2.69 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.59E+01. Sample per second: 14871\n",
      "MF_BPR: Epoch 799 of 1500. Elapsed time 2.69 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 1.59E+01. Sample per second: 13759\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.25 sec. Users per second: 1785\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.04 sec. Users per second: 1784\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2314063, PRECISION: 0.0839492, PRECISION_RECALL_MIN_DEN: 0.1063396, RECALL: 0.0691249, MAP: 0.0488097, MRR: 0.1928101, NDCG: 0.0676616, F1: 0.0758192, HIT_RATE: 0.8394916, ARHR: 0.2719110, NOVELTY: 0.0097530, AVERAGE_POPULARITY: 0.4834465, DIVERSITY_MEAN_INTER_LIST: 0.9154530, DIVERSITY_HERFINDAHL: 0.9915440, COVERAGE_ITEM: 0.9872671, COVERAGE_ITEM_CORRECT: 0.0617920, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4184293, DIVERSITY_GINI: 0.1276600, SHANNON_ENTROPY: 8.9249712, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 800 of 1500. Elapsed time 2.70 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.43 seconds. BPR loss 1.59E+01. Sample per second: 15774\n",
      "MF_BPR: Epoch 801 of 1500. Elapsed time 2.70 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 1.58E+01. Sample per second: 15090\n",
      "MF_BPR: Epoch 802 of 1500. Elapsed time 2.71 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.60E+01. Sample per second: 14476\n",
      "MF_BPR: Epoch 803 of 1500. Elapsed time 2.71 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 1.58E+01. Sample per second: 13940\n",
      "MF_BPR: Epoch 804 of 1500. Elapsed time 2.71 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 1.58E+01. Sample per second: 16496\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.35 sec. Users per second: 1779\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.84 sec. Users per second: 1793\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2315377, PRECISION: 0.0839621, PRECISION_RECALL_MIN_DEN: 0.1063429, RECALL: 0.0691467, MAP: 0.0488237, MRR: 0.1929712, NDCG: 0.0677247, F1: 0.0758376, HIT_RATE: 0.8396209, ARHR: 0.2720113, NOVELTY: 0.0097525, AVERAGE_POPULARITY: 0.4830813, DIVERSITY_MEAN_INTER_LIST: 0.9155610, DIVERSITY_HERFINDAHL: 0.9915548, COVERAGE_ITEM: 0.9870799, COVERAGE_ITEM_CORRECT: 0.0616047, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4186582, DIVERSITY_GINI: 0.1273049, SHANNON_ENTROPY: 8.9255175, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 805 of 1500. Elapsed time 2.72 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.59E+01. Sample per second: 16108\n",
      "MF_BPR: Epoch 806 of 1500. Elapsed time 2.72 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 1.59E+01. Sample per second: 15586\n",
      "MF_BPR: Epoch 807 of 1500. Elapsed time 2.72 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.73 seconds. BPR loss 1.59E+01. Sample per second: 14785\n",
      "MF_BPR: Epoch 808 of 1500. Elapsed time 2.72 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.98 seconds. BPR loss 1.60E+01. Sample per second: 14025\n",
      "MF_BPR: Epoch 809 of 1500. Elapsed time 2.72 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.18 seconds. BPR loss 1.58E+01. Sample per second: 13481\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.68% ) in 30.35 sec. Users per second: 1713\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.34 sec. Users per second: 1726\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2316784, PRECISION: 0.0840755, PRECISION_RECALL_MIN_DEN: 0.1064824, RECALL: 0.0692461, MAP: 0.0489393, MRR: 0.1933164, NDCG: 0.0678636, F1: 0.0759437, HIT_RATE: 0.8407554, ARHR: 0.2724651, NOVELTY: 0.0097524, AVERAGE_POPULARITY: 0.4825893, DIVERSITY_MEAN_INTER_LIST: 0.9158691, DIVERSITY_HERFINDAHL: 0.9915856, COVERAGE_ITEM: 0.9869862, COVERAGE_ITEM_CORRECT: 0.0616047, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4191734, DIVERSITY_GINI: 0.1269807, SHANNON_ENTROPY: 8.9278803, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 810 of 1500. Elapsed time 2.74 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 1.60E+01. Sample per second: 14488\n",
      "MF_BPR: Epoch 811 of 1500. Elapsed time 2.74 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 1.60E+01. Sample per second: 13945\n",
      "MF_BPR: Epoch 812 of 1500. Elapsed time 2.74 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.60E+01. Sample per second: 16084\n",
      "MF_BPR: Epoch 813 of 1500. Elapsed time 2.74 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 1.60E+01. Sample per second: 14536\n",
      "MF_BPR: Epoch 814 of 1500. Elapsed time 2.74 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.13 seconds. BPR loss 1.60E+01. Sample per second: 13626\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.49 sec. Users per second: 1804\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.28 sec. Users per second: 1819\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2315759, PRECISION: 0.0841761, PRECISION_RECALL_MIN_DEN: 0.1066636, RECALL: 0.0694020, MAP: 0.0489974, MRR: 0.1933754, NDCG: 0.0679232, F1: 0.0760784, HIT_RATE: 0.8417606, ARHR: 0.2726475, NOVELTY: 0.0097518, AVERAGE_POPULARITY: 0.4820452, DIVERSITY_MEAN_INTER_LIST: 0.9163615, DIVERSITY_HERFINDAHL: 0.9916348, COVERAGE_ITEM: 0.9867990, COVERAGE_ITEM_CORRECT: 0.0624473, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4192020, DIVERSITY_GINI: 0.1267121, SHANNON_ENTROPY: 8.9312207, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 815 of 1500. Elapsed time 2.75 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 1.61E+01. Sample per second: 14333\n",
      "MF_BPR: Epoch 816 of 1500. Elapsed time 2.75 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.23 seconds. BPR loss 1.61E+01. Sample per second: 13371\n",
      "MF_BPR: Epoch 817 of 1500. Elapsed time 2.76 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 1.60E+01. Sample per second: 15618\n",
      "MF_BPR: Epoch 818 of 1500. Elapsed time 2.76 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 1.62E+01. Sample per second: 14799\n",
      "MF_BPR: Epoch 819 of 1500. Elapsed time 2.76 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 1.62E+01. Sample per second: 14280\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.39 sec. Users per second: 1810\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.68 sec. Users per second: 1755\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2315469, PRECISION: 0.0841962, PRECISION_RECALL_MIN_DEN: 0.1066577, RECALL: 0.0693602, MAP: 0.0489820, MRR: 0.1932770, NDCG: 0.0678817, F1: 0.0760615, HIT_RATE: 0.8419617, ARHR: 0.2726275, NOVELTY: 0.0097504, AVERAGE_POPULARITY: 0.4819801, DIVERSITY_MEAN_INTER_LIST: 0.9165282, DIVERSITY_HERFINDAHL: 0.9916515, COVERAGE_ITEM: 0.9867054, COVERAGE_ITEM_CORRECT: 0.0627282, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4191734, DIVERSITY_GINI: 0.1264276, SHANNON_ENTROPY: 8.9314145, \n",
      "\n",
      "MF_BPR: Epoch 820 of 1500. Elapsed time 2.77 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 1.61E+01. Sample per second: 14139\n",
      "MF_BPR: Epoch 821 of 1500. Elapsed time 2.77 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.62E+01. Sample per second: 13851\n",
      "MF_BPR: Epoch 822 of 1500. Elapsed time 2.77 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.15 seconds. BPR loss 1.63E+01. Sample per second: 16823\n",
      "MF_BPR: Epoch 823 of 1500. Elapsed time 2.77 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.27 seconds. BPR loss 1.62E+01. Sample per second: 16383\n",
      "MF_BPR: Epoch 824 of 1500. Elapsed time 2.77 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 1.62E+01. Sample per second: 15924\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.17 sec. Users per second: 1757\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.28 sec. Users per second: 1773\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2316374, PRECISION: 0.0842953, PRECISION_RECALL_MIN_DEN: 0.1068204, RECALL: 0.0695142, MAP: 0.0490725, MRR: 0.1935074, NDCG: 0.0679845, F1: 0.0761945, HIT_RATE: 0.8429525, ARHR: 0.2730010, NOVELTY: 0.0097502, AVERAGE_POPULARITY: 0.4815975, DIVERSITY_MEAN_INTER_LIST: 0.9167770, DIVERSITY_HERFINDAHL: 0.9916764, COVERAGE_ITEM: 0.9866117, COVERAGE_ITEM_CORRECT: 0.0625410, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4193451, DIVERSITY_GINI: 0.1260607, SHANNON_ENTROPY: 8.9331261, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 825 of 1500. Elapsed time 2.79 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.62E+01. Sample per second: 13714\n",
      "MF_BPR: Epoch 826 of 1500. Elapsed time 2.79 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.62E+01. Sample per second: 16318\n",
      "MF_BPR: Epoch 827 of 1500. Elapsed time 2.79 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.62E+01. Sample per second: 13886\n",
      "MF_BPR: Epoch 828 of 1500. Elapsed time 2.79 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 1.62E+01. Sample per second: 15981\n",
      "MF_BPR: Epoch 829 of 1500. Elapsed time 2.79 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.63E+01. Sample per second: 15418\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.54 sec. Users per second: 1801\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.11 sec. Users per second: 1781\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2320853, PRECISION: 0.0842493, PRECISION_RECALL_MIN_DEN: 0.1067654, RECALL: 0.0694732, MAP: 0.0491285, MRR: 0.1937424, NDCG: 0.0679853, F1: 0.0761511, HIT_RATE: 0.8424930, ARHR: 0.2732029, NOVELTY: 0.0097494, AVERAGE_POPULARITY: 0.4814255, DIVERSITY_MEAN_INTER_LIST: 0.9169698, DIVERSITY_HERFINDAHL: 0.9916957, COVERAGE_ITEM: 0.9867990, COVERAGE_ITEM_CORRECT: 0.0627282, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4193451, DIVERSITY_GINI: 0.1257927, SHANNON_ENTROPY: 8.9343147, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 830 of 1500. Elapsed time 2.80 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 1.62E+01. Sample per second: 14109\n",
      "MF_BPR: Epoch 831 of 1500. Elapsed time 2.80 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.13 seconds. BPR loss 1.62E+01. Sample per second: 13632\n",
      "MF_BPR: Epoch 832 of 1500. Elapsed time 2.81 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 1.62E+01. Sample per second: 15926\n",
      "MF_BPR: Epoch 833 of 1500. Elapsed time 2.81 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 1.64E+01. Sample per second: 15196\n",
      "MF_BPR: Epoch 834 of 1500. Elapsed time 2.81 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.27 seconds. BPR loss 1.62E+01. Sample per second: 13268\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.46 sec. Users per second: 1806\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.44 sec. Users per second: 1811\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2319571, PRECISION: 0.0843541, PRECISION_RECALL_MIN_DEN: 0.1068880, RECALL: 0.0695434, MAP: 0.0491710, MRR: 0.1938372, NDCG: 0.0679558, F1: 0.0762361, HIT_RATE: 0.8435413, ARHR: 0.2734280, NOVELTY: 0.0097496, AVERAGE_POPULARITY: 0.4808893, DIVERSITY_MEAN_INTER_LIST: 0.9173817, DIVERSITY_HERFINDAHL: 0.9917369, COVERAGE_ITEM: 0.9861436, COVERAGE_ITEM_CORRECT: 0.0629155, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4196886, DIVERSITY_GINI: 0.1255368, SHANNON_ENTROPY: 8.9377455, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 835 of 1500. Elapsed time 2.82 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 1.63E+01. Sample per second: 14118\n",
      "MF_BPR: Epoch 836 of 1500. Elapsed time 2.82 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.11 seconds. BPR loss 1.64E+01. Sample per second: 13668\n",
      "MF_BPR: Epoch 837 of 1500. Elapsed time 2.82 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.23 seconds. BPR loss 1.63E+01. Sample per second: 16532\n",
      "MF_BPR: Epoch 838 of 1500. Elapsed time 2.82 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 1.63E+01. Sample per second: 15902\n",
      "MF_BPR: Epoch 839 of 1500. Elapsed time 2.82 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 1.64E+01. Sample per second: 15286\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.41 sec. Users per second: 1775\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.07 sec. Users per second: 1782\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2320856, PRECISION: 0.0843771, PRECISION_RECALL_MIN_DEN: 0.1069640, RECALL: 0.0696124, MAP: 0.0491873, MRR: 0.1939131, NDCG: 0.0679853, F1: 0.0762870, HIT_RATE: 0.8437711, ARHR: 0.2735087, NOVELTY: 0.0097484, AVERAGE_POPULARITY: 0.4807899, DIVERSITY_MEAN_INTER_LIST: 0.9174559, DIVERSITY_HERFINDAHL: 0.9917443, COVERAGE_ITEM: 0.9858627, COVERAGE_ITEM_CORRECT: 0.0627282, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4200034, DIVERSITY_GINI: 0.1252293, SHANNON_ENTROPY: 8.9383715, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 840 of 1500. Elapsed time 2.84 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 1.63E+01. Sample per second: 14306\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 841 of 1500. Elapsed time 2.84 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.65E+01. Sample per second: 13892\n",
      "MF_BPR: Epoch 842 of 1500. Elapsed time 2.84 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 1.64E+01. Sample per second: 16603\n",
      "MF_BPR: Epoch 843 of 1500. Elapsed time 2.84 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 1.64E+01. Sample per second: 15687\n",
      "MF_BPR: Epoch 844 of 1500. Elapsed time 2.84 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 1.64E+01. Sample per second: 15298\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.03 sec. Users per second: 1798\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.77 sec. Users per second: 1796\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2320063, PRECISION: 0.0843096, PRECISION_RECALL_MIN_DEN: 0.1069213, RECALL: 0.0696438, MAP: 0.0491262, MRR: 0.1937598, NDCG: 0.0679503, F1: 0.0762782, HIT_RATE: 0.8430961, ARHR: 0.2732670, NOVELTY: 0.0097493, AVERAGE_POPULARITY: 0.4801061, DIVERSITY_MEAN_INTER_LIST: 0.9178275, DIVERSITY_HERFINDAHL: 0.9917814, COVERAGE_ITEM: 0.9860500, COVERAGE_ITEM_CORRECT: 0.0629155, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4202038, DIVERSITY_GINI: 0.1250170, SHANNON_ENTROPY: 8.9413226, \n",
      "\n",
      "MF_BPR: Epoch 845 of 1500. Elapsed time 2.85 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 1.64E+01. Sample per second: 15083\n",
      "MF_BPR: Epoch 846 of 1500. Elapsed time 2.85 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 1.65E+01. Sample per second: 14323\n",
      "MF_BPR: Epoch 847 of 1500. Elapsed time 2.86 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 1.65E+01. Sample per second: 13857\n",
      "MF_BPR: Epoch 848 of 1500. Elapsed time 2.86 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 1.65E+01. Sample per second: 16293\n",
      "MF_BPR: Epoch 849 of 1500. Elapsed time 2.86 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 1.65E+01. Sample per second: 15138\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 53000 ( 76.11% ) in 30.49 sec. Users per second: 1738\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.59 sec. Users per second: 1759\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2319360, PRECISION: 0.0843570, PRECISION_RECALL_MIN_DEN: 0.1069795, RECALL: 0.0696469, MAP: 0.0491531, MRR: 0.1936761, NDCG: 0.0678850, F1: 0.0762994, HIT_RATE: 0.8435700, ARHR: 0.2732352, NOVELTY: 0.0097481, AVERAGE_POPULARITY: 0.4801456, DIVERSITY_MEAN_INTER_LIST: 0.9178113, DIVERSITY_HERFINDAHL: 0.9917798, COVERAGE_ITEM: 0.9858627, COVERAGE_ITEM_CORRECT: 0.0626346, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4201752, DIVERSITY_GINI: 0.1247168, SHANNON_ENTROPY: 8.9401045, \n",
      "\n",
      "MF_BPR: Epoch 850 of 1500. Elapsed time 2.87 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 1.64E+01. Sample per second: 15494\n",
      "MF_BPR: Epoch 851 of 1500. Elapsed time 2.87 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 1.66E+01. Sample per second: 14920\n",
      "MF_BPR: Epoch 852 of 1500. Elapsed time 2.87 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 1.65E+01. Sample per second: 14311\n",
      "MF_BPR: Epoch 853 of 1500. Elapsed time 2.87 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.20 seconds. BPR loss 1.65E+01. Sample per second: 13439\n",
      "MF_BPR: Epoch 854 of 1500. Elapsed time 2.88 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 1.66E+01. Sample per second: 16042\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.98% ) in 30.28 sec. Users per second: 1817\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 38.16 sec. Users per second: 1825\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2323409, PRECISION: 0.0842479, PRECISION_RECALL_MIN_DEN: 0.1068607, RECALL: 0.0695935, MAP: 0.0491233, MRR: 0.1938033, NDCG: 0.0678517, F1: 0.0762227, HIT_RATE: 0.8424786, ARHR: 0.2732030, NOVELTY: 0.0097477, AVERAGE_POPULARITY: 0.4797915, DIVERSITY_MEAN_INTER_LIST: 0.9179555, DIVERSITY_HERFINDAHL: 0.9917942, COVERAGE_ITEM: 0.9855819, COVERAGE_ITEM_CORRECT: 0.0627282, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4201609, DIVERSITY_GINI: 0.1244605, SHANNON_ENTROPY: 8.9409689, \n",
      "\n",
      "MF_BPR: Epoch 855 of 1500. Elapsed time 2.89 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 1.65E+01. Sample per second: 14451\n",
      "MF_BPR: Epoch 856 of 1500. Elapsed time 2.89 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.36 seconds. BPR loss 1.65E+01. Sample per second: 13047\n",
      "MF_BPR: Epoch 857 of 1500. Elapsed time 2.89 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 1.66E+01. Sample per second: 15580\n",
      "MF_BPR: Epoch 858 of 1500. Elapsed time 2.89 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 1.66E+01. Sample per second: 14817\n",
      "MF_BPR: Epoch 859 of 1500. Elapsed time 2.89 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.67E+01. Sample per second: 14219\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.55% ) in 30.52 sec. Users per second: 1769\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 39.23 sec. Users per second: 1775\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2323119, PRECISION: 0.0843211, PRECISION_RECALL_MIN_DEN: 0.1068758, RECALL: 0.0695511, MAP: 0.0491485, MRR: 0.1937889, NDCG: 0.0678787, F1: 0.0762273, HIT_RATE: 0.8432110, ARHR: 0.2733055, NOVELTY: 0.0097471, AVERAGE_POPULARITY: 0.4798605, DIVERSITY_MEAN_INTER_LIST: 0.9179466, DIVERSITY_HERFINDAHL: 0.9917933, COVERAGE_ITEM: 0.9859564, COVERAGE_ITEM_CORRECT: 0.0626346, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4200607, DIVERSITY_GINI: 0.1241861, SHANNON_ENTROPY: 8.9411741, \n",
      "\n",
      "MF_BPR: Epoch 860 of 1500. Elapsed time 2.90 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 1.67E+01. Sample per second: 15697\n",
      "MF_BPR: Epoch 861 of 1500. Elapsed time 2.90 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.67E+01. Sample per second: 14865\n",
      "MF_BPR: Epoch 862 of 1500. Elapsed time 2.91 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.22 seconds. BPR loss 1.67E+01. Sample per second: 13397\n",
      "MF_BPR: Epoch 863 of 1500. Elapsed time 2.91 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 1.66E+01. Sample per second: 16080\n",
      "MF_BPR: Epoch 864 of 1500. Elapsed time 2.91 hour\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 1.67E+01. Sample per second: 14974\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 51000 ( 73.24% ) in 30.16 sec. Users per second: 1691\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 40.28 sec. Users per second: 1729\n",
      "MF_BPR: CUTOFF: 10 - ROC_AUC: 0.2322616, PRECISION: 0.0842421, PRECISION_RECALL_MIN_DEN: 0.1068190, RECALL: 0.0695496, MAP: 0.0490874, MRR: 0.1936516, NDCG: 0.0678286, F1: 0.0761940, HIT_RATE: 0.8424212, ARHR: 0.2730524, NOVELTY: 0.0097468, AVERAGE_POPULARITY: 0.4794515, DIVERSITY_MEAN_INTER_LIST: 0.9182329, DIVERSITY_HERFINDAHL: 0.9918220, COVERAGE_ITEM: 0.9858627, COVERAGE_ITEM_CORRECT: 0.0625410, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4200034, DIVERSITY_GINI: 0.1239227, SHANNON_ENTROPY: 8.9430454, \n",
      "\n",
      "MF_BPR: Convergence reached! Terminating at epoch 865. Best value for 'MAP' at epoch 840 is 0.0492. Elapsed time 2.92 hour\n",
      "MF_BPR: Epoch 865 of 1500. Elapsed time 2.92 hour\n",
      "EvaluatorHoldout: Processed 57000 ( 81.86% ) in 30.45 sec. Users per second: 1872\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 37.39 sec. Users per second: 1863\n",
      "SearchBayesianSkopt: New best config found. Config 4: {'sgd_mode': 'adam', 'epochs': 840, 'num_factors': 200, 'batch_size': 1, 'positive_reg': 2.1264549316860416e-05, 'negative_reg': 0.00033305173444924584, 'learning_rate': 0.00010151223842175566} - results: ROC_AUC: 0.2320856, PRECISION: 0.0843771, PRECISION_RECALL_MIN_DEN: 0.1069640, RECALL: 0.0696124, MAP: 0.0491873, MRR: 0.1939131, NDCG: 0.0679853, F1: 0.0762870, HIT_RATE: 0.8437711, ARHR: 0.2735087, NOVELTY: 0.0097484, AVERAGE_POPULARITY: 0.4807899, DIVERSITY_MEAN_INTER_LIST: 0.9174559, DIVERSITY_HERFINDAHL: 0.9917443, COVERAGE_ITEM: 0.9858627, COVERAGE_ITEM_CORRECT: 0.0627282, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.4200034, DIVERSITY_GINI: 0.1252293, SHANNON_ENTROPY: 8.9383715, \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 54000 ( 77.38% ) in 30.26 sec. Users per second: 1784\n",
      "EvaluatorHoldout: Processed 69786 ( 100.00% ) in 39.65 sec. Users per second: 1760\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test. Config: {'sgd_mode': 'adam', 'epochs': 840, 'num_factors': 200, 'batch_size': 1, 'positive_reg': 2.1264549316860416e-05, 'negative_reg': 0.00033305173444924584, 'learning_rate': 0.00010151223842175566} - results:\n",
      "CUTOFF: 10 - ROC_AUC: 0.2560357, PRECISION: 0.1056788, PRECISION_RECALL_MIN_DEN: 0.1223342, RECALL: 0.0689357, MAP: 0.0614934, MRR: 0.2285226, NDCG: 0.0732627, F1: 0.0834414, HIT_RATE: 1.0567879, ARHR: 0.3436550, NOVELTY: 0.0097476, AVERAGE_POPULARITY: 0.4808200, DIVERSITY_MEAN_INTER_LIST: 0.9174522, DIVERSITY_HERFINDAHL: 0.9917439, COVERAGE_ITEM: 0.9858627, COVERAGE_ITEM_CORRECT: 0.0674094, COVERAGE_USER: 0.9986834, COVERAGE_USER_CORRECT: 0.4605885, DIVERSITY_GINI: 0.1251575, SHANNON_ENTROPY: 8.9376416, \n",
      "\n",
      "\n",
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 10594.1313\n",
      "Function value obtained: -0.0492\n",
      "Current minimum: -0.0492\n",
      "SearchBayesianSkopt: Search complete. Best config is 4: {'sgd_mode': 'adam', 'epochs': 840, 'num_factors': 200, 'batch_size': 1, 'positive_reg': 2.1264549316860416e-05, 'negative_reg': 0.00033305173444924584, 'learning_rate': 0.00010151223842175566}\n",
      "\n",
      "MatrixFactorization_BPR_Cython_Recommender: URM Detected 37 (0.35 %) cold items.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'sgd_mode': 'adam', 'epochs': 840, 'num_factors': 200, 'batch_size': 1, 'positive_reg': 2.1264549316860416e-05, 'negative_reg': 0.00033305173444924584, 'learning_rate': 0.00010151223842175566}\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.37 seconds. BPR loss 4.01E-02. Sample per second: 13024\n",
      "MF_BPR: Epoch 1 of 840. Elapsed time 4.55 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 4.04E-02. Sample per second: 15062\n",
      "MF_BPR: Epoch 2 of 840. Elapsed time 8.82 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 4.00E-02. Sample per second: 14351\n",
      "MF_BPR: Epoch 3 of 840. Elapsed time 13.05 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 3.97E-02. Sample per second: 13729\n",
      "MF_BPR: Epoch 4 of 840. Elapsed time 17.27 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 4.01E-02. Sample per second: 16168\n",
      "MF_BPR: Epoch 5 of 840. Elapsed time 21.51 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 4.01E-02. Sample per second: 15165\n",
      "MF_BPR: Epoch 6 of 840. Elapsed time 25.79 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.21 seconds. BPR loss 3.99E-02. Sample per second: 13414\n",
      "MF_BPR: Epoch 7 of 840. Elapsed time 30.39 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 3.95E-02. Sample per second: 15838\n",
      "MF_BPR: Epoch 8 of 840. Elapsed time 34.60 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 4.02E-02. Sample per second: 14956\n",
      "MF_BPR: Epoch 9 of 840. Elapsed time 38.86 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 4.00E-02. Sample per second: 14167\n",
      "MF_BPR: Epoch 10 of 840. Elapsed time 43.12 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.22 seconds. BPR loss 3.99E-02. Sample per second: 13396\n",
      "MF_BPR: Epoch 11 of 840. Elapsed time 47.40 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 4.00E-02. Sample per second: 15509\n",
      "MF_BPR: Epoch 12 of 840. Elapsed time 51.69 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 4.00E-02. Sample per second: 14720\n",
      "MF_BPR: Epoch 13 of 840. Elapsed time 55.93 sec\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 3.98E-02. Sample per second: 14130\n",
      "MF_BPR: Epoch 14 of 840. Elapsed time 1.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 4.00E-02. Sample per second: 13652\n",
      "MF_BPR: Epoch 15 of 840. Elapsed time 1.07 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 4.00E-02. Sample per second: 16186\n",
      "MF_BPR: Epoch 16 of 840. Elapsed time 1.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 4.00E-02. Sample per second: 15482\n",
      "MF_BPR: Epoch 17 of 840. Elapsed time 1.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 3.96E-02. Sample per second: 14830\n",
      "MF_BPR: Epoch 18 of 840. Elapsed time 1.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 3.99E-02. Sample per second: 14171\n",
      "MF_BPR: Epoch 19 of 840. Elapsed time 1.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.33 seconds. BPR loss 4.02E-02. Sample per second: 13115\n",
      "MF_BPR: Epoch 20 of 840. Elapsed time 1.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 3.96E-02. Sample per second: 15130\n",
      "MF_BPR: Epoch 21 of 840. Elapsed time 1.50 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 4.01E-02. Sample per second: 14298\n",
      "MF_BPR: Epoch 22 of 840. Elapsed time 1.57 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 4.01E-02. Sample per second: 13814\n",
      "MF_BPR: Epoch 23 of 840. Elapsed time 1.64 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 4.00E-02. Sample per second: 16571\n",
      "MF_BPR: Epoch 24 of 840. Elapsed time 1.71 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 4.00E-02. Sample per second: 15598\n",
      "MF_BPR: Epoch 25 of 840. Elapsed time 1.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 4.02E-02. Sample per second: 14951\n",
      "MF_BPR: Epoch 26 of 840. Elapsed time 1.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 4.02E-02. Sample per second: 14420\n",
      "MF_BPR: Epoch 27 of 840. Elapsed time 1.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 4.00E-02. Sample per second: 13799\n",
      "MF_BPR: Epoch 28 of 840. Elapsed time 1.99 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 4.04E-02. Sample per second: 16158\n",
      "MF_BPR: Epoch 29 of 840. Elapsed time 2.06 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 3.99E-02. Sample per second: 15386\n",
      "MF_BPR: Epoch 30 of 840. Elapsed time 2.13 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 4.04E-02. Sample per second: 14640\n",
      "MF_BPR: Epoch 31 of 840. Elapsed time 2.20 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 4.05E-02. Sample per second: 13738\n",
      "MF_BPR: Epoch 32 of 840. Elapsed time 2.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 4.06E-02. Sample per second: 15845\n",
      "MF_BPR: Epoch 33 of 840. Elapsed time 2.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 4.09E-02. Sample per second: 14835\n",
      "MF_BPR: Epoch 34 of 840. Elapsed time 2.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 4.03E-02. Sample per second: 14226\n",
      "MF_BPR: Epoch 35 of 840. Elapsed time 2.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.17 seconds. BPR loss 4.02E-02. Sample per second: 13504\n",
      "MF_BPR: Epoch 36 of 840. Elapsed time 2.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 4.06E-02. Sample per second: 14987\n",
      "MF_BPR: Epoch 37 of 840. Elapsed time 2.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 4.07E-02. Sample per second: 13575\n",
      "MF_BPR: Epoch 38 of 840. Elapsed time 2.71 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 4.04E-02. Sample per second: 15990\n",
      "MF_BPR: Epoch 39 of 840. Elapsed time 2.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 4.09E-02. Sample per second: 15223\n",
      "MF_BPR: Epoch 40 of 840. Elapsed time 2.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 4.14E-02. Sample per second: 14234\n",
      "MF_BPR: Epoch 41 of 840. Elapsed time 2.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 4.07E-02. Sample per second: 13563\n",
      "MF_BPR: Epoch 42 of 840. Elapsed time 2.99 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 4.13E-02. Sample per second: 15601\n",
      "MF_BPR: Epoch 43 of 840. Elapsed time 3.06 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 4.16E-02. Sample per second: 14722\n",
      "MF_BPR: Epoch 44 of 840. Elapsed time 3.13 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 4.14E-02. Sample per second: 13942\n",
      "MF_BPR: Epoch 45 of 840. Elapsed time 3.20 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.27 seconds. BPR loss 4.12E-02. Sample per second: 16384\n",
      "MF_BPR: Epoch 46 of 840. Elapsed time 3.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 4.15E-02. Sample per second: 15294\n",
      "MF_BPR: Epoch 47 of 840. Elapsed time 3.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 4.18E-02. Sample per second: 14325\n",
      "MF_BPR: Epoch 48 of 840. Elapsed time 3.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 4.18E-02. Sample per second: 13790\n",
      "MF_BPR: Epoch 49 of 840. Elapsed time 3.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 4.22E-02. Sample per second: 15812\n",
      "MF_BPR: Epoch 50 of 840. Elapsed time 3.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 4.20E-02. Sample per second: 15012\n",
      "MF_BPR: Epoch 51 of 840. Elapsed time 3.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 4.25E-02. Sample per second: 14343\n",
      "MF_BPR: Epoch 52 of 840. Elapsed time 3.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.20 seconds. BPR loss 4.24E-02. Sample per second: 13440\n",
      "MF_BPR: Epoch 53 of 840. Elapsed time 3.77 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 4.22E-02. Sample per second: 15187\n",
      "MF_BPR: Epoch 54 of 840. Elapsed time 3.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 4.27E-02. Sample per second: 14407\n",
      "MF_BPR: Epoch 55 of 840. Elapsed time 3.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.11 seconds. BPR loss 4.32E-02. Sample per second: 13664\n",
      "MF_BPR: Epoch 56 of 840. Elapsed time 3.99 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 4.31E-02. Sample per second: 16071\n",
      "MF_BPR: Epoch 57 of 840. Elapsed time 4.06 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 4.33E-02. Sample per second: 15315\n",
      "MF_BPR: Epoch 58 of 840. Elapsed time 4.13 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 4.35E-02. Sample per second: 14556\n",
      "MF_BPR: Epoch 59 of 840. Elapsed time 4.20 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 4.36E-02. Sample per second: 13872\n",
      "MF_BPR: Epoch 60 of 840. Elapsed time 4.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 4.40E-02. Sample per second: 16158\n",
      "MF_BPR: Epoch 61 of 840. Elapsed time 4.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 4.42E-02. Sample per second: 15358\n",
      "MF_BPR: Epoch 62 of 840. Elapsed time 4.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 4.47E-02. Sample per second: 14589\n",
      "MF_BPR: Epoch 63 of 840. Elapsed time 4.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.20 seconds. BPR loss 4.47E-02. Sample per second: 13433\n",
      "MF_BPR: Epoch 64 of 840. Elapsed time 4.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 4.50E-02. Sample per second: 15642\n",
      "MF_BPR: Epoch 65 of 840. Elapsed time 4.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 4.55E-02. Sample per second: 14105\n",
      "MF_BPR: Epoch 66 of 840. Elapsed time 4.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.62 seconds. BPR loss 4.53E-02. Sample per second: 12430\n",
      "MF_BPR: Epoch 67 of 840. Elapsed time 4.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.24 seconds. BPR loss 4.57E-02. Sample per second: 13343\n",
      "MF_BPR: Epoch 68 of 840. Elapsed time 4.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 4.60E-02. Sample per second: 14415\n",
      "MF_BPR: Epoch 69 of 840. Elapsed time 4.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.19 seconds. BPR loss 4.71E-02. Sample per second: 13467\n",
      "MF_BPR: Epoch 70 of 840. Elapsed time 5.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 4.70E-02. Sample per second: 15232\n",
      "MF_BPR: Epoch 71 of 840. Elapsed time 5.08 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 4.77E-02. Sample per second: 13795\n",
      "MF_BPR: Epoch 72 of 840. Elapsed time 5.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 4.84E-02. Sample per second: 15420\n",
      "MF_BPR: Epoch 73 of 840. Elapsed time 5.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 4.85E-02. Sample per second: 14155\n",
      "MF_BPR: Epoch 74 of 840. Elapsed time 5.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 4.92E-02. Sample per second: 13554\n",
      "MF_BPR: Epoch 75 of 840. Elapsed time 5.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 4.98E-02. Sample per second: 16021\n",
      "MF_BPR: Epoch 76 of 840. Elapsed time 5.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 4.96E-02. Sample per second: 15391\n",
      "MF_BPR: Epoch 77 of 840. Elapsed time 5.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 5.09E-02. Sample per second: 14484\n",
      "MF_BPR: Epoch 78 of 840. Elapsed time 5.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 5.10E-02. Sample per second: 13803\n",
      "MF_BPR: Epoch 79 of 840. Elapsed time 5.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 5.23E-02. Sample per second: 16216\n",
      "MF_BPR: Epoch 80 of 840. Elapsed time 5.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 5.31E-02. Sample per second: 15127\n",
      "MF_BPR: Epoch 81 of 840. Elapsed time 5.80 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 5.40E-02. Sample per second: 13812\n",
      "MF_BPR: Epoch 82 of 840. Elapsed time 5.87 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 5.45E-02. Sample per second: 14392\n",
      "MF_BPR: Epoch 83 of 840. Elapsed time 5.95 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.44 seconds. BPR loss 5.48E-02. Sample per second: 12851\n",
      "MF_BPR: Epoch 84 of 840. Elapsed time 6.03 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 5.61E-02. Sample per second: 14710\n",
      "MF_BPR: Epoch 85 of 840. Elapsed time 6.10 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 5.70E-02. Sample per second: 13871\n",
      "MF_BPR: Epoch 86 of 840. Elapsed time 6.17 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 5.78E-02. Sample per second: 16581\n",
      "MF_BPR: Epoch 87 of 840. Elapsed time 6.24 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 5.95E-02. Sample per second: 15946\n",
      "MF_BPR: Epoch 88 of 840. Elapsed time 6.31 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 6.08E-02. Sample per second: 15327\n",
      "MF_BPR: Epoch 89 of 840. Elapsed time 6.38 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 6.15E-02. Sample per second: 14707\n",
      "MF_BPR: Epoch 90 of 840. Elapsed time 6.45 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 6.24E-02. Sample per second: 14066\n",
      "MF_BPR: Epoch 91 of 840. Elapsed time 6.52 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.23 seconds. BPR loss 6.40E-02. Sample per second: 13356\n",
      "MF_BPR: Epoch 92 of 840. Elapsed time 6.59 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 6.61E-02. Sample per second: 15657\n",
      "MF_BPR: Epoch 93 of 840. Elapsed time 6.66 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 6.68E-02. Sample per second: 14917\n",
      "MF_BPR: Epoch 94 of 840. Elapsed time 6.73 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 6.87E-02. Sample per second: 14228\n",
      "MF_BPR: Epoch 95 of 840. Elapsed time 6.80 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 7.05E-02. Sample per second: 13720\n",
      "MF_BPR: Epoch 96 of 840. Elapsed time 6.87 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.25 seconds. BPR loss 7.27E-02. Sample per second: 16456\n",
      "MF_BPR: Epoch 97 of 840. Elapsed time 6.94 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 7.48E-02. Sample per second: 15804\n",
      "MF_BPR: Epoch 98 of 840. Elapsed time 7.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 7.54E-02. Sample per second: 15053\n",
      "MF_BPR: Epoch 99 of 840. Elapsed time 7.08 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 7.91E-02. Sample per second: 14529\n",
      "MF_BPR: Epoch 100 of 840. Elapsed time 7.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 8.04E-02. Sample per second: 13967\n",
      "MF_BPR: Epoch 101 of 840. Elapsed time 7.22 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.19 seconds. BPR loss 8.33E-02. Sample per second: 16695\n",
      "MF_BPR: Epoch 102 of 840. Elapsed time 7.29 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 8.42E-02. Sample per second: 15568\n",
      "MF_BPR: Epoch 103 of 840. Elapsed time 7.36 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 8.66E-02. Sample per second: 14470\n",
      "MF_BPR: Epoch 104 of 840. Elapsed time 7.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.22 seconds. BPR loss 9.13E-02. Sample per second: 13387\n",
      "MF_BPR: Epoch 105 of 840. Elapsed time 7.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 9.51E-02. Sample per second: 15367\n",
      "MF_BPR: Epoch 106 of 840. Elapsed time 7.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 9.55E-02. Sample per second: 13983\n",
      "MF_BPR: Epoch 107 of 840. Elapsed time 7.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 9.88E-02. Sample per second: 16035\n",
      "MF_BPR: Epoch 108 of 840. Elapsed time 7.73 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 1.03E-01. Sample per second: 15000\n",
      "MF_BPR: Epoch 109 of 840. Elapsed time 7.80 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.73 seconds. BPR loss 1.06E-01. Sample per second: 14788\n",
      "MF_BPR: Epoch 110 of 840. Elapsed time 7.87 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 1.10E-01. Sample per second: 14608\n",
      "MF_BPR: Epoch 111 of 840. Elapsed time 7.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 1.14E-01. Sample per second: 13727\n",
      "MF_BPR: Epoch 112 of 840. Elapsed time 8.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 1.18E-01. Sample per second: 16472\n",
      "MF_BPR: Epoch 113 of 840. Elapsed time 8.07 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 1.21E-01. Sample per second: 15869\n",
      "MF_BPR: Epoch 114 of 840. Elapsed time 8.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 1.26E-01. Sample per second: 15137\n",
      "MF_BPR: Epoch 115 of 840. Elapsed time 8.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 1.29E-01. Sample per second: 14433\n",
      "MF_BPR: Epoch 116 of 840. Elapsed time 8.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.11 seconds. BPR loss 1.35E-01. Sample per second: 13681\n",
      "MF_BPR: Epoch 117 of 840. Elapsed time 8.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 1.40E-01. Sample per second: 16182\n",
      "MF_BPR: Epoch 118 of 840. Elapsed time 8.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.43E-01. Sample per second: 15542\n",
      "MF_BPR: Epoch 119 of 840. Elapsed time 8.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.51E-01. Sample per second: 14557\n",
      "MF_BPR: Epoch 120 of 840. Elapsed time 8.57 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 1.58E-01. Sample per second: 13947\n",
      "MF_BPR: Epoch 121 of 840. Elapsed time 8.64 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.10 seconds. BPR loss 1.62E-01. Sample per second: 17041\n",
      "MF_BPR: Epoch 122 of 840. Elapsed time 8.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 1.68E-01. Sample per second: 16195\n",
      "MF_BPR: Epoch 123 of 840. Elapsed time 8.77 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.52 seconds. BPR loss 1.77E-01. Sample per second: 15443\n",
      "MF_BPR: Epoch 124 of 840. Elapsed time 8.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 1.86E-01. Sample per second: 14505\n",
      "MF_BPR: Epoch 125 of 840. Elapsed time 8.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.39 seconds. BPR loss 1.90E-01. Sample per second: 12967\n",
      "MF_BPR: Epoch 126 of 840. Elapsed time 8.99 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 1.97E-01. Sample per second: 14710\n",
      "MF_BPR: Epoch 127 of 840. Elapsed time 9.07 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 2.05E-01. Sample per second: 14067\n",
      "MF_BPR: Epoch 128 of 840. Elapsed time 9.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.13 seconds. BPR loss 2.09E-01. Sample per second: 13609\n",
      "MF_BPR: Epoch 129 of 840. Elapsed time 9.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.08 seconds. BPR loss 2.18E-01. Sample per second: 17106\n",
      "MF_BPR: Epoch 130 of 840. Elapsed time 9.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 2.28E-01. Sample per second: 16295\n",
      "MF_BPR: Epoch 131 of 840. Elapsed time 9.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 2.38E-01. Sample per second: 15996\n",
      "MF_BPR: Epoch 132 of 840. Elapsed time 9.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 2.50E-01. Sample per second: 15503\n",
      "MF_BPR: Epoch 133 of 840. Elapsed time 9.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 2.54E-01. Sample per second: 15548\n",
      "MF_BPR: Epoch 134 of 840. Elapsed time 9.54 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 2.64E-01. Sample per second: 15103\n",
      "MF_BPR: Epoch 135 of 840. Elapsed time 9.61 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 2.75E-01. Sample per second: 14519\n",
      "MF_BPR: Epoch 136 of 840. Elapsed time 9.68 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 2.84E-01. Sample per second: 14320\n",
      "MF_BPR: Epoch 137 of 840. Elapsed time 9.75 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.24 seconds. BPR loss 2.94E-01. Sample per second: 13342\n",
      "MF_BPR: Epoch 138 of 840. Elapsed time 9.82 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 3.06E-01. Sample per second: 16479\n",
      "MF_BPR: Epoch 139 of 840. Elapsed time 9.89 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.23 seconds. BPR loss 3.15E-01. Sample per second: 16503\n",
      "MF_BPR: Epoch 140 of 840. Elapsed time 9.96 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 3.30E-01. Sample per second: 16277\n",
      "MF_BPR: Epoch 141 of 840. Elapsed time 10.02 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 3.40E-01. Sample per second: 15688\n",
      "MF_BPR: Epoch 142 of 840. Elapsed time 10.09 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 3.50E-01. Sample per second: 15331\n",
      "MF_BPR: Epoch 143 of 840. Elapsed time 10.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 3.72E-01. Sample per second: 15298\n",
      "MF_BPR: Epoch 144 of 840. Elapsed time 10.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 3.77E-01. Sample per second: 14930\n",
      "MF_BPR: Epoch 145 of 840. Elapsed time 10.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 3.93E-01. Sample per second: 14214\n",
      "MF_BPR: Epoch 146 of 840. Elapsed time 10.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.11 seconds. BPR loss 4.07E-01. Sample per second: 13662\n",
      "MF_BPR: Epoch 147 of 840. Elapsed time 10.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 4.07E-01. Sample per second: 16589\n",
      "MF_BPR: Epoch 148 of 840. Elapsed time 10.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 4.35E-01. Sample per second: 16176\n",
      "MF_BPR: Epoch 149 of 840. Elapsed time 10.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 4.46E-01. Sample per second: 15409\n",
      "MF_BPR: Epoch 150 of 840. Elapsed time 10.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 4.65E-01. Sample per second: 15343\n",
      "MF_BPR: Epoch 151 of 840. Elapsed time 10.71 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 4.80E-01. Sample per second: 15058\n",
      "MF_BPR: Epoch 152 of 840. Elapsed time 10.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 4.94E-01. Sample per second: 14274\n",
      "MF_BPR: Epoch 153 of 840. Elapsed time 10.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 5.14E-01. Sample per second: 13910\n",
      "MF_BPR: Epoch 154 of 840. Elapsed time 10.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.11 seconds. BPR loss 5.30E-01. Sample per second: 16984\n",
      "MF_BPR: Epoch 155 of 840. Elapsed time 10.99 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.02 seconds. BPR loss 5.45E-01. Sample per second: 17375\n",
      "MF_BPR: Epoch 156 of 840. Elapsed time 11.05 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.52 seconds. BPR loss 5.63E-01. Sample per second: 15454\n",
      "MF_BPR: Epoch 157 of 840. Elapsed time 11.13 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 5.89E-01. Sample per second: 14958\n",
      "MF_BPR: Epoch 158 of 840. Elapsed time 11.20 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 5.90E-01. Sample per second: 14446\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 159 of 840. Elapsed time 11.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 6.19E-01. Sample per second: 13904\n",
      "MF_BPR: Epoch 160 of 840. Elapsed time 11.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 6.31E-01. Sample per second: 16113\n",
      "MF_BPR: Epoch 161 of 840. Elapsed time 11.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.52 seconds. BPR loss 6.51E-01. Sample per second: 15454\n",
      "MF_BPR: Epoch 162 of 840. Elapsed time 11.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 6.78E-01. Sample per second: 14917\n",
      "MF_BPR: Epoch 163 of 840. Elapsed time 11.55 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 6.90E-01. Sample per second: 14364\n",
      "MF_BPR: Epoch 164 of 840. Elapsed time 11.62 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 7.10E-01. Sample per second: 13886\n",
      "MF_BPR: Epoch 165 of 840. Elapsed time 11.69 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.27 seconds. BPR loss 7.39E-01. Sample per second: 16347\n",
      "MF_BPR: Epoch 166 of 840. Elapsed time 11.76 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 7.51E-01. Sample per second: 15585\n",
      "MF_BPR: Epoch 167 of 840. Elapsed time 11.83 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 7.64E-01. Sample per second: 14927\n",
      "MF_BPR: Epoch 168 of 840. Elapsed time 11.90 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 7.96E-01. Sample per second: 14366\n",
      "MF_BPR: Epoch 169 of 840. Elapsed time 11.97 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 8.05E-01. Sample per second: 13824\n",
      "MF_BPR: Epoch 170 of 840. Elapsed time 12.04 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.26 seconds. BPR loss 8.28E-01. Sample per second: 16404\n",
      "MF_BPR: Epoch 171 of 840. Elapsed time 12.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 8.65E-01. Sample per second: 15642\n",
      "MF_BPR: Epoch 172 of 840. Elapsed time 12.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 8.85E-01. Sample per second: 14980\n",
      "MF_BPR: Epoch 173 of 840. Elapsed time 12.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 9.05E-01. Sample per second: 14471\n",
      "MF_BPR: Epoch 174 of 840. Elapsed time 12.32 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 9.39E-01. Sample per second: 13954\n",
      "MF_BPR: Epoch 175 of 840. Elapsed time 12.39 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.19 seconds. BPR loss 9.49E-01. Sample per second: 16670\n",
      "MF_BPR: Epoch 176 of 840. Elapsed time 12.46 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 9.78E-01. Sample per second: 15704\n",
      "MF_BPR: Epoch 177 of 840. Elapsed time 12.53 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 1.00E+00. Sample per second: 15127\n",
      "MF_BPR: Epoch 178 of 840. Elapsed time 12.60 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 1.03E+00. Sample per second: 14641\n",
      "MF_BPR: Epoch 179 of 840. Elapsed time 12.67 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.05E+00. Sample per second: 14228\n",
      "MF_BPR: Epoch 180 of 840. Elapsed time 12.73 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.07E+00. Sample per second: 13808\n",
      "MF_BPR: Epoch 181 of 840. Elapsed time 12.80 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 1.11E+00. Sample per second: 16486\n",
      "MF_BPR: Epoch 182 of 840. Elapsed time 12.87 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 1.13E+00. Sample per second: 15974\n",
      "MF_BPR: Epoch 183 of 840. Elapsed time 12.94 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 1.16E+00. Sample per second: 15393\n",
      "MF_BPR: Epoch 184 of 840. Elapsed time 13.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 1.19E+00. Sample per second: 14805\n",
      "MF_BPR: Epoch 185 of 840. Elapsed time 13.08 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 1.21E+00. Sample per second: 14177\n",
      "MF_BPR: Epoch 186 of 840. Elapsed time 13.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.24E+00. Sample per second: 13640\n",
      "MF_BPR: Epoch 187 of 840. Elapsed time 13.22 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 1.26E+00. Sample per second: 16259\n",
      "MF_BPR: Epoch 188 of 840. Elapsed time 13.29 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 1.29E+00. Sample per second: 15637\n",
      "MF_BPR: Epoch 189 of 840. Elapsed time 13.36 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 1.32E+00. Sample per second: 15028\n",
      "MF_BPR: Epoch 190 of 840. Elapsed time 13.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 1.35E+00. Sample per second: 14430\n",
      "MF_BPR: Epoch 191 of 840. Elapsed time 13.50 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 1.38E+00. Sample per second: 13936\n",
      "MF_BPR: Epoch 192 of 840. Elapsed time 13.57 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.15 seconds. BPR loss 1.40E+00. Sample per second: 16823\n",
      "MF_BPR: Epoch 193 of 840. Elapsed time 13.64 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 1.44E+00. Sample per second: 16241\n",
      "MF_BPR: Epoch 194 of 840. Elapsed time 13.71 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 1.48E+00. Sample per second: 15742\n",
      "MF_BPR: Epoch 195 of 840. Elapsed time 13.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.58 seconds. BPR loss 1.52E+00. Sample per second: 15261\n",
      "MF_BPR: Epoch 196 of 840. Elapsed time 13.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.73 seconds. BPR loss 1.52E+00. Sample per second: 14786\n",
      "MF_BPR: Epoch 197 of 840. Elapsed time 13.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 1.55E+00. Sample per second: 14361\n",
      "MF_BPR: Epoch 198 of 840. Elapsed time 13.98 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 1.59E+00. Sample per second: 13910\n",
      "MF_BPR: Epoch 199 of 840. Elapsed time 14.05 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.19 seconds. BPR loss 1.61E+00. Sample per second: 16674\n",
      "MF_BPR: Epoch 200 of 840. Elapsed time 14.12 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 1.67E+00. Sample per second: 16050\n",
      "MF_BPR: Epoch 201 of 840. Elapsed time 14.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 1.69E+00. Sample per second: 15382\n",
      "MF_BPR: Epoch 202 of 840. Elapsed time 14.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 1.71E+00. Sample per second: 14717\n",
      "MF_BPR: Epoch 203 of 840. Elapsed time 14.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 1.74E+00. Sample per second: 13965\n",
      "MF_BPR: Epoch 204 of 840. Elapsed time 14.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 1.78E+00. Sample per second: 15946\n",
      "MF_BPR: Epoch 205 of 840. Elapsed time 14.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 1.81E+00. Sample per second: 14910\n",
      "MF_BPR: Epoch 206 of 840. Elapsed time 14.55 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 1.85E+00. Sample per second: 14194\n",
      "MF_BPR: Epoch 207 of 840. Elapsed time 14.62 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.11 seconds. BPR loss 1.88E+00. Sample per second: 13667\n",
      "MF_BPR: Epoch 208 of 840. Elapsed time 14.69 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 1.91E+00. Sample per second: 16023\n",
      "MF_BPR: Epoch 209 of 840. Elapsed time 14.76 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.96E+00. Sample per second: 15419\n",
      "MF_BPR: Epoch 210 of 840. Elapsed time 14.83 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.96E+00. Sample per second: 14872\n",
      "MF_BPR: Epoch 211 of 840. Elapsed time 14.90 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 2.01E+00. Sample per second: 14370\n",
      "MF_BPR: Epoch 212 of 840. Elapsed time 14.97 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 2.02E+00. Sample per second: 13848\n",
      "MF_BPR: Epoch 213 of 840. Elapsed time 15.04 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 2.05E+00. Sample per second: 16500\n",
      "MF_BPR: Epoch 214 of 840. Elapsed time 15.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 2.10E+00. Sample per second: 15840\n",
      "MF_BPR: Epoch 215 of 840. Elapsed time 15.18 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 2.12E+00. Sample per second: 15037\n",
      "MF_BPR: Epoch 216 of 840. Elapsed time 15.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 2.15E+00. Sample per second: 14581\n",
      "MF_BPR: Epoch 217 of 840. Elapsed time 15.32 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 2.21E+00. Sample per second: 14202\n",
      "MF_BPR: Epoch 218 of 840. Elapsed time 15.39 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 2.22E+00. Sample per second: 13805\n",
      "MF_BPR: Epoch 219 of 840. Elapsed time 15.45 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.19 seconds. BPR loss 2.27E+00. Sample per second: 16670\n",
      "MF_BPR: Epoch 220 of 840. Elapsed time 15.52 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 2.28E+00. Sample per second: 16072\n",
      "MF_BPR: Epoch 221 of 840. Elapsed time 15.59 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 2.35E+00. Sample per second: 15551\n",
      "MF_BPR: Epoch 222 of 840. Elapsed time 15.66 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 2.37E+00. Sample per second: 15012\n",
      "MF_BPR: Epoch 223 of 840. Elapsed time 15.73 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 2.41E+00. Sample per second: 14443\n",
      "MF_BPR: Epoch 224 of 840. Elapsed time 15.80 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 2.45E+00. Sample per second: 13939\n",
      "MF_BPR: Epoch 225 of 840. Elapsed time 15.87 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.15 seconds. BPR loss 2.52E+00. Sample per second: 16841\n",
      "MF_BPR: Epoch 226 of 840. Elapsed time 15.94 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 2.54E+00. Sample per second: 16236\n",
      "MF_BPR: Epoch 227 of 840. Elapsed time 16.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 2.54E+00. Sample per second: 15625\n",
      "MF_BPR: Epoch 228 of 840. Elapsed time 16.08 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 2.55E+00. Sample per second: 15111\n",
      "MF_BPR: Epoch 229 of 840. Elapsed time 16.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 2.64E+00. Sample per second: 14611\n",
      "MF_BPR: Epoch 230 of 840. Elapsed time 16.22 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 2.67E+00. Sample per second: 14163\n",
      "MF_BPR: Epoch 231 of 840. Elapsed time 16.29 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 2.72E+00. Sample per second: 13691\n",
      "MF_BPR: Epoch 232 of 840. Elapsed time 16.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.25 seconds. BPR loss 2.76E+00. Sample per second: 16427\n",
      "MF_BPR: Epoch 233 of 840. Elapsed time 16.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 2.75E+00. Sample per second: 15813\n",
      "MF_BPR: Epoch 234 of 840. Elapsed time 16.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 2.82E+00. Sample per second: 15178\n",
      "MF_BPR: Epoch 235 of 840. Elapsed time 16.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 2.84E+00. Sample per second: 14573\n",
      "MF_BPR: Epoch 236 of 840. Elapsed time 16.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 2.90E+00. Sample per second: 13972\n",
      "MF_BPR: Epoch 237 of 840. Elapsed time 16.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.23 seconds. BPR loss 2.92E+00. Sample per second: 16520\n",
      "MF_BPR: Epoch 238 of 840. Elapsed time 16.77 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 2.93E+00. Sample per second: 15727\n",
      "MF_BPR: Epoch 239 of 840. Elapsed time 16.84 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 2.99E+00. Sample per second: 15116\n",
      "MF_BPR: Epoch 240 of 840. Elapsed time 16.91 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 3.02E+00. Sample per second: 14600\n",
      "MF_BPR: Epoch 241 of 840. Elapsed time 16.98 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 3.07E+00. Sample per second: 14177\n",
      "MF_BPR: Epoch 242 of 840. Elapsed time 17.05 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 3.12E+00. Sample per second: 13816\n",
      "MF_BPR: Epoch 243 of 840. Elapsed time 17.12 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 3.12E+00. Sample per second: 16574\n",
      "MF_BPR: Epoch 244 of 840. Elapsed time 17.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 3.15E+00. Sample per second: 15888\n",
      "MF_BPR: Epoch 245 of 840. Elapsed time 17.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 3.17E+00. Sample per second: 15293\n",
      "MF_BPR: Epoch 246 of 840. Elapsed time 17.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 3.31E+00. Sample per second: 14835\n",
      "MF_BPR: Epoch 247 of 840. Elapsed time 17.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 3.28E+00. Sample per second: 14351\n",
      "MF_BPR: Epoch 248 of 840. Elapsed time 17.47 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 3.32E+00. Sample per second: 13966\n",
      "MF_BPR: Epoch 249 of 840. Elapsed time 17.54 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.15 seconds. BPR loss 3.38E+00. Sample per second: 16833\n",
      "MF_BPR: Epoch 250 of 840. Elapsed time 17.61 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 3.42E+00. Sample per second: 16217\n",
      "MF_BPR: Epoch 251 of 840. Elapsed time 17.67 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 3.41E+00. Sample per second: 15630\n",
      "MF_BPR: Epoch 252 of 840. Elapsed time 17.74 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 3.50E+00. Sample per second: 15109\n",
      "MF_BPR: Epoch 253 of 840. Elapsed time 17.81 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 3.50E+00. Sample per second: 14202\n",
      "MF_BPR: Epoch 254 of 840. Elapsed time 17.89 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 3.55E+00. Sample per second: 13636\n",
      "MF_BPR: Epoch 255 of 840. Elapsed time 17.96 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 3.58E+00. Sample per second: 16151\n",
      "MF_BPR: Epoch 256 of 840. Elapsed time 18.03 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 3.66E+00. Sample per second: 15527\n",
      "MF_BPR: Epoch 257 of 840. Elapsed time 18.09 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 3.65E+00. Sample per second: 14991\n",
      "MF_BPR: Epoch 258 of 840. Elapsed time 18.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 3.68E+00. Sample per second: 14413\n",
      "MF_BPR: Epoch 259 of 840. Elapsed time 18.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 3.75E+00. Sample per second: 13911\n",
      "MF_BPR: Epoch 260 of 840. Elapsed time 18.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 3.77E+00. Sample per second: 16562\n",
      "MF_BPR: Epoch 261 of 840. Elapsed time 18.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 3.82E+00. Sample per second: 15741\n",
      "MF_BPR: Epoch 262 of 840. Elapsed time 18.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 3.91E+00. Sample per second: 15089\n",
      "MF_BPR: Epoch 263 of 840. Elapsed time 18.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 3.90E+00. Sample per second: 14578\n",
      "MF_BPR: Epoch 264 of 840. Elapsed time 18.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.96 seconds. BPR loss 3.97E+00. Sample per second: 14096\n",
      "MF_BPR: Epoch 265 of 840. Elapsed time 18.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.11 seconds. BPR loss 3.97E+00. Sample per second: 13663\n",
      "MF_BPR: Epoch 266 of 840. Elapsed time 18.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 4.06E+00. Sample per second: 16338\n",
      "MF_BPR: Epoch 267 of 840. Elapsed time 18.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 4.05E+00. Sample per second: 15720\n",
      "MF_BPR: Epoch 268 of 840. Elapsed time 18.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 4.08E+00. Sample per second: 15105\n",
      "MF_BPR: Epoch 269 of 840. Elapsed time 18.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 4.14E+00. Sample per second: 14588\n",
      "MF_BPR: Epoch 270 of 840. Elapsed time 19.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 4.14E+00. Sample per second: 14132\n",
      "MF_BPR: Epoch 271 of 840. Elapsed time 19.07 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 4.23E+00. Sample per second: 13636\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 272 of 840. Elapsed time 19.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 4.21E+00. Sample per second: 16204\n",
      "MF_BPR: Epoch 273 of 840. Elapsed time 19.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 4.29E+00. Sample per second: 15075\n",
      "MF_BPR: Epoch 274 of 840. Elapsed time 19.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 4.33E+00. Sample per second: 14555\n",
      "MF_BPR: Epoch 275 of 840. Elapsed time 19.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.98 seconds. BPR loss 4.36E+00. Sample per second: 14019\n",
      "MF_BPR: Epoch 276 of 840. Elapsed time 19.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.17 seconds. BPR loss 4.43E+00. Sample per second: 13525\n",
      "MF_BPR: Epoch 277 of 840. Elapsed time 19.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 4.46E+00. Sample per second: 15653\n",
      "MF_BPR: Epoch 278 of 840. Elapsed time 19.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.73 seconds. BPR loss 4.52E+00. Sample per second: 14784\n",
      "MF_BPR: Epoch 279 of 840. Elapsed time 19.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 4.55E+00. Sample per second: 14051\n",
      "MF_BPR: Epoch 280 of 840. Elapsed time 19.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.18 seconds. BPR loss 4.59E+00. Sample per second: 13479\n",
      "MF_BPR: Epoch 281 of 840. Elapsed time 19.77 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 4.61E+00. Sample per second: 15891\n",
      "MF_BPR: Epoch 282 of 840. Elapsed time 19.84 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 4.61E+00. Sample per second: 15140\n",
      "MF_BPR: Epoch 283 of 840. Elapsed time 19.91 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 4.67E+00. Sample per second: 14444\n",
      "MF_BPR: Epoch 284 of 840. Elapsed time 19.98 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 4.71E+00. Sample per second: 13831\n",
      "MF_BPR: Epoch 285 of 840. Elapsed time 20.05 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 4.82E+00. Sample per second: 16146\n",
      "MF_BPR: Epoch 286 of 840. Elapsed time 20.13 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 4.78E+00. Sample per second: 14189\n",
      "MF_BPR: Epoch 287 of 840. Elapsed time 20.20 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 4.87E+00. Sample per second: 13562\n",
      "MF_BPR: Epoch 288 of 840. Elapsed time 20.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 4.89E+00. Sample per second: 16205\n",
      "MF_BPR: Epoch 289 of 840. Elapsed time 20.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 4.92E+00. Sample per second: 15566\n",
      "MF_BPR: Epoch 290 of 840. Elapsed time 20.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 4.94E+00. Sample per second: 14970\n",
      "MF_BPR: Epoch 291 of 840. Elapsed time 20.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 4.99E+00. Sample per second: 14320\n",
      "MF_BPR: Epoch 292 of 840. Elapsed time 20.55 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 4.95E+00. Sample per second: 13842\n",
      "MF_BPR: Epoch 293 of 840. Elapsed time 20.62 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.20 seconds. BPR loss 5.03E+00. Sample per second: 16646\n",
      "MF_BPR: Epoch 294 of 840. Elapsed time 20.69 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 5.06E+00. Sample per second: 15971\n",
      "MF_BPR: Epoch 295 of 840. Elapsed time 20.76 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 5.09E+00. Sample per second: 15389\n",
      "MF_BPR: Epoch 296 of 840. Elapsed time 20.83 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 5.15E+00. Sample per second: 14737\n",
      "MF_BPR: Epoch 297 of 840. Elapsed time 20.90 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 5.19E+00. Sample per second: 14243\n",
      "MF_BPR: Epoch 298 of 840. Elapsed time 20.97 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 5.27E+00. Sample per second: 13759\n",
      "MF_BPR: Epoch 299 of 840. Elapsed time 21.04 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.25 seconds. BPR loss 5.33E+00. Sample per second: 16447\n",
      "MF_BPR: Epoch 300 of 840. Elapsed time 21.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 5.30E+00. Sample per second: 15935\n",
      "MF_BPR: Epoch 301 of 840. Elapsed time 21.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 5.34E+00. Sample per second: 15103\n",
      "MF_BPR: Epoch 302 of 840. Elapsed time 21.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 5.39E+00. Sample per second: 14638\n",
      "MF_BPR: Epoch 303 of 840. Elapsed time 21.32 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 5.48E+00. Sample per second: 13847\n",
      "MF_BPR: Epoch 304 of 840. Elapsed time 21.39 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 5.51E+00. Sample per second: 16262\n",
      "MF_BPR: Epoch 305 of 840. Elapsed time 21.46 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 5.61E+00. Sample per second: 15098\n",
      "MF_BPR: Epoch 306 of 840. Elapsed time 21.53 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 5.59E+00. Sample per second: 14378\n",
      "MF_BPR: Epoch 307 of 840. Elapsed time 21.60 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.21 seconds. BPR loss 5.62E+00. Sample per second: 13422\n",
      "MF_BPR: Epoch 308 of 840. Elapsed time 21.67 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 5.62E+00. Sample per second: 15991\n",
      "MF_BPR: Epoch 309 of 840. Elapsed time 21.74 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 5.61E+00. Sample per second: 15062\n",
      "MF_BPR: Epoch 310 of 840. Elapsed time 21.81 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 5.65E+00. Sample per second: 14548\n",
      "MF_BPR: Epoch 311 of 840. Elapsed time 21.88 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.98 seconds. BPR loss 5.73E+00. Sample per second: 14031\n",
      "MF_BPR: Epoch 312 of 840. Elapsed time 21.95 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.14 seconds. BPR loss 5.75E+00. Sample per second: 13608\n",
      "MF_BPR: Epoch 313 of 840. Elapsed time 22.02 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 5.88E+00. Sample per second: 16271\n",
      "MF_BPR: Epoch 314 of 840. Elapsed time 22.09 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 5.79E+00. Sample per second: 15687\n",
      "MF_BPR: Epoch 315 of 840. Elapsed time 22.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 5.93E+00. Sample per second: 15128\n",
      "MF_BPR: Epoch 316 of 840. Elapsed time 22.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 5.91E+00. Sample per second: 14663\n",
      "MF_BPR: Epoch 317 of 840. Elapsed time 22.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 5.96E+00. Sample per second: 14196\n",
      "MF_BPR: Epoch 318 of 840. Elapsed time 22.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 6.02E+00. Sample per second: 13754\n",
      "MF_BPR: Epoch 319 of 840. Elapsed time 22.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 6.01E+00. Sample per second: 16463\n",
      "MF_BPR: Epoch 320 of 840. Elapsed time 22.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 6.05E+00. Sample per second: 15883\n",
      "MF_BPR: Epoch 321 of 840. Elapsed time 22.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 6.12E+00. Sample per second: 15308\n",
      "MF_BPR: Epoch 322 of 840. Elapsed time 22.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.73 seconds. BPR loss 6.11E+00. Sample per second: 14785\n",
      "MF_BPR: Epoch 323 of 840. Elapsed time 22.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 6.27E+00. Sample per second: 13914\n",
      "MF_BPR: Epoch 324 of 840. Elapsed time 22.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 6.19E+00. Sample per second: 15841\n",
      "MF_BPR: Epoch 325 of 840. Elapsed time 22.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 6.23E+00. Sample per second: 15000\n",
      "MF_BPR: Epoch 326 of 840. Elapsed time 22.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 6.27E+00. Sample per second: 14352\n",
      "MF_BPR: Epoch 327 of 840. Elapsed time 23.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 6.29E+00. Sample per second: 13541\n",
      "MF_BPR: Epoch 328 of 840. Elapsed time 23.07 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 6.38E+00. Sample per second: 15638\n",
      "MF_BPR: Epoch 329 of 840. Elapsed time 23.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 6.35E+00. Sample per second: 14670\n",
      "MF_BPR: Epoch 330 of 840. Elapsed time 23.22 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.96 seconds. BPR loss 6.44E+00. Sample per second: 14087\n",
      "MF_BPR: Epoch 331 of 840. Elapsed time 23.29 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.25 seconds. BPR loss 6.50E+00. Sample per second: 13304\n",
      "MF_BPR: Epoch 332 of 840. Elapsed time 23.36 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 6.47E+00. Sample per second: 15422\n",
      "MF_BPR: Epoch 333 of 840. Elapsed time 23.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 6.51E+00. Sample per second: 13878\n",
      "MF_BPR: Epoch 334 of 840. Elapsed time 23.50 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 6.58E+00. Sample per second: 15700\n",
      "MF_BPR: Epoch 335 of 840. Elapsed time 23.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 6.61E+00. Sample per second: 14381\n",
      "MF_BPR: Epoch 336 of 840. Elapsed time 23.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.21 seconds. BPR loss 6.68E+00. Sample per second: 13410\n",
      "MF_BPR: Epoch 337 of 840. Elapsed time 23.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 6.67E+00. Sample per second: 15653\n",
      "MF_BPR: Epoch 338 of 840. Elapsed time 23.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 6.76E+00. Sample per second: 14904\n",
      "MF_BPR: Epoch 339 of 840. Elapsed time 23.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 6.76E+00. Sample per second: 14127\n",
      "MF_BPR: Epoch 340 of 840. Elapsed time 23.94 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 6.83E+00. Sample per second: 13530\n",
      "MF_BPR: Epoch 341 of 840. Elapsed time 24.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 6.87E+00. Sample per second: 15644\n",
      "MF_BPR: Epoch 342 of 840. Elapsed time 24.08 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 6.91E+00. Sample per second: 14684\n",
      "MF_BPR: Epoch 343 of 840. Elapsed time 24.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.14 seconds. BPR loss 6.87E+00. Sample per second: 13596\n",
      "MF_BPR: Epoch 344 of 840. Elapsed time 24.22 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 6.86E+00. Sample per second: 15309\n",
      "MF_BPR: Epoch 345 of 840. Elapsed time 24.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 6.91E+00. Sample per second: 14563\n",
      "MF_BPR: Epoch 346 of 840. Elapsed time 24.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 7.00E+00. Sample per second: 13877\n",
      "MF_BPR: Epoch 347 of 840. Elapsed time 24.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 7.01E+00. Sample per second: 16109\n",
      "MF_BPR: Epoch 348 of 840. Elapsed time 24.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 7.10E+00. Sample per second: 15130\n",
      "MF_BPR: Epoch 349 of 840. Elapsed time 24.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 7.10E+00. Sample per second: 14415\n",
      "MF_BPR: Epoch 350 of 840. Elapsed time 24.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 7.17E+00. Sample per second: 13835\n",
      "MF_BPR: Epoch 351 of 840. Elapsed time 24.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 7.22E+00. Sample per second: 16246\n",
      "MF_BPR: Epoch 352 of 840. Elapsed time 24.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 7.21E+00. Sample per second: 15008\n",
      "MF_BPR: Epoch 353 of 840. Elapsed time 24.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 7.23E+00. Sample per second: 14217\n",
      "MF_BPR: Epoch 354 of 840. Elapsed time 24.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 7.25E+00. Sample per second: 13639\n",
      "MF_BPR: Epoch 355 of 840. Elapsed time 25.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 7.25E+00. Sample per second: 15875\n",
      "MF_BPR: Epoch 356 of 840. Elapsed time 25.08 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 7.33E+00. Sample per second: 14974\n",
      "MF_BPR: Epoch 357 of 840. Elapsed time 25.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 7.29E+00. Sample per second: 14264\n",
      "MF_BPR: Epoch 358 of 840. Elapsed time 25.22 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.18 seconds. BPR loss 7.37E+00. Sample per second: 13500\n",
      "MF_BPR: Epoch 359 of 840. Elapsed time 25.29 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 7.43E+00. Sample per second: 15846\n",
      "MF_BPR: Epoch 360 of 840. Elapsed time 25.36 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 7.50E+00. Sample per second: 14809\n",
      "MF_BPR: Epoch 361 of 840. Elapsed time 25.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 7.43E+00. Sample per second: 13946\n",
      "MF_BPR: Epoch 362 of 840. Elapsed time 25.50 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 7.59E+00. Sample per second: 16274\n",
      "MF_BPR: Epoch 363 of 840. Elapsed time 25.57 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 7.56E+00. Sample per second: 15314\n",
      "MF_BPR: Epoch 364 of 840. Elapsed time 25.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 7.55E+00. Sample per second: 14261\n",
      "MF_BPR: Epoch 365 of 840. Elapsed time 25.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.24 seconds. BPR loss 7.62E+00. Sample per second: 13344\n",
      "MF_BPR: Epoch 366 of 840. Elapsed time 25.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 7.59E+00. Sample per second: 15495\n",
      "MF_BPR: Epoch 367 of 840. Elapsed time 25.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 7.61E+00. Sample per second: 14288\n",
      "MF_BPR: Epoch 368 of 840. Elapsed time 25.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.27 seconds. BPR loss 7.72E+00. Sample per second: 13268\n",
      "MF_BPR: Epoch 369 of 840. Elapsed time 26.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 7.73E+00. Sample per second: 15307\n",
      "MF_BPR: Epoch 370 of 840. Elapsed time 26.08 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 7.79E+00. Sample per second: 14367\n",
      "MF_BPR: Epoch 371 of 840. Elapsed time 26.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 7.78E+00. Sample per second: 13540\n",
      "MF_BPR: Epoch 372 of 840. Elapsed time 26.22 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 7.80E+00. Sample per second: 15890\n",
      "MF_BPR: Epoch 373 of 840. Elapsed time 26.29 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 7.84E+00. Sample per second: 14973\n",
      "MF_BPR: Epoch 374 of 840. Elapsed time 26.36 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.96 seconds. BPR loss 7.97E+00. Sample per second: 14090\n",
      "MF_BPR: Epoch 375 of 840. Elapsed time 26.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.20 seconds. BPR loss 7.93E+00. Sample per second: 13449\n",
      "MF_BPR: Epoch 376 of 840. Elapsed time 26.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 7.95E+00. Sample per second: 15880\n",
      "MF_BPR: Epoch 377 of 840. Elapsed time 26.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 7.99E+00. Sample per second: 14962\n",
      "MF_BPR: Epoch 378 of 840. Elapsed time 26.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 8.07E+00. Sample per second: 14135\n",
      "MF_BPR: Epoch 379 of 840. Elapsed time 26.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.17 seconds. BPR loss 8.03E+00. Sample per second: 13528\n",
      "MF_BPR: Epoch 380 of 840. Elapsed time 26.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 8.01E+00. Sample per second: 16063\n",
      "MF_BPR: Epoch 381 of 840. Elapsed time 26.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 8.22E+00. Sample per second: 15430\n",
      "MF_BPR: Epoch 382 of 840. Elapsed time 26.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 8.31E+00. Sample per second: 14798\n",
      "MF_BPR: Epoch 383 of 840. Elapsed time 27.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 8.16E+00. Sample per second: 14217\n",
      "MF_BPR: Epoch 384 of 840. Elapsed time 27.07 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 8.31E+00. Sample per second: 13703\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 385 of 840. Elapsed time 27.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 8.21E+00. Sample per second: 16342\n",
      "MF_BPR: Epoch 386 of 840. Elapsed time 27.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 8.25E+00. Sample per second: 15711\n",
      "MF_BPR: Epoch 387 of 840. Elapsed time 27.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.73 seconds. BPR loss 8.32E+00. Sample per second: 14761\n",
      "MF_BPR: Epoch 388 of 840. Elapsed time 27.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 8.30E+00. Sample per second: 14005\n",
      "MF_BPR: Epoch 389 of 840. Elapsed time 27.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.22 seconds. BPR loss 8.31E+00. Sample per second: 13392\n",
      "MF_BPR: Epoch 390 of 840. Elapsed time 27.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 8.40E+00. Sample per second: 15366\n",
      "MF_BPR: Epoch 391 of 840. Elapsed time 27.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 8.39E+00. Sample per second: 14223\n",
      "MF_BPR: Epoch 392 of 840. Elapsed time 27.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.19 seconds. BPR loss 8.45E+00. Sample per second: 13451\n",
      "MF_BPR: Epoch 393 of 840. Elapsed time 27.71 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 8.59E+00. Sample per second: 15806\n",
      "MF_BPR: Epoch 394 of 840. Elapsed time 27.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 8.51E+00. Sample per second: 14902\n",
      "MF_BPR: Epoch 395 of 840. Elapsed time 27.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 8.64E+00. Sample per second: 14313\n",
      "MF_BPR: Epoch 396 of 840. Elapsed time 27.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 8.52E+00. Sample per second: 13651\n",
      "MF_BPR: Epoch 397 of 840. Elapsed time 27.99 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 8.73E+00. Sample per second: 15948\n",
      "MF_BPR: Epoch 398 of 840. Elapsed time 28.06 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.63 seconds. BPR loss 8.72E+00. Sample per second: 15100\n",
      "MF_BPR: Epoch 399 of 840. Elapsed time 28.13 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 8.63E+00. Sample per second: 14311\n",
      "MF_BPR: Epoch 400 of 840. Elapsed time 28.20 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.16 seconds. BPR loss 8.74E+00. Sample per second: 13556\n",
      "MF_BPR: Epoch 401 of 840. Elapsed time 28.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 8.68E+00. Sample per second: 15554\n",
      "MF_BPR: Epoch 402 of 840. Elapsed time 28.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 8.78E+00. Sample per second: 14530\n",
      "MF_BPR: Epoch 403 of 840. Elapsed time 28.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 8.79E+00. Sample per second: 13744\n",
      "MF_BPR: Epoch 404 of 840. Elapsed time 28.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 8.81E+00. Sample per second: 16169\n",
      "MF_BPR: Epoch 405 of 840. Elapsed time 28.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 8.74E+00. Sample per second: 15416\n",
      "MF_BPR: Epoch 406 of 840. Elapsed time 28.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 8.87E+00. Sample per second: 14684\n",
      "MF_BPR: Epoch 407 of 840. Elapsed time 28.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 8.95E+00. Sample per second: 14137\n",
      "MF_BPR: Epoch 408 of 840. Elapsed time 28.77 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 8.92E+00. Sample per second: 13660\n",
      "MF_BPR: Epoch 409 of 840. Elapsed time 28.84 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 8.94E+00. Sample per second: 16133\n",
      "MF_BPR: Epoch 410 of 840. Elapsed time 28.91 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 8.96E+00. Sample per second: 15153\n",
      "MF_BPR: Epoch 411 of 840. Elapsed time 28.98 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 9.02E+00. Sample per second: 14429\n",
      "MF_BPR: Epoch 412 of 840. Elapsed time 29.05 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 9.04E+00. Sample per second: 13932\n",
      "MF_BPR: Epoch 413 of 840. Elapsed time 29.12 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 9.08E+00. Sample per second: 16607\n",
      "MF_BPR: Epoch 414 of 840. Elapsed time 29.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 9.00E+00. Sample per second: 15985\n",
      "MF_BPR: Epoch 415 of 840. Elapsed time 29.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 9.20E+00. Sample per second: 15426\n",
      "MF_BPR: Epoch 416 of 840. Elapsed time 29.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 9.11E+00. Sample per second: 14862\n",
      "MF_BPR: Epoch 417 of 840. Elapsed time 29.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 9.20E+00. Sample per second: 14379\n",
      "MF_BPR: Epoch 418 of 840. Elapsed time 29.47 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.02 seconds. BPR loss 9.19E+00. Sample per second: 13915\n",
      "MF_BPR: Epoch 419 of 840. Elapsed time 29.54 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.19 seconds. BPR loss 9.21E+00. Sample per second: 16676\n",
      "MF_BPR: Epoch 420 of 840. Elapsed time 29.61 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 9.32E+00. Sample per second: 15996\n",
      "MF_BPR: Epoch 421 of 840. Elapsed time 29.68 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 9.26E+00. Sample per second: 15354\n",
      "MF_BPR: Epoch 422 of 840. Elapsed time 29.75 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 9.32E+00. Sample per second: 14790\n",
      "MF_BPR: Epoch 423 of 840. Elapsed time 29.82 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 9.28E+00. Sample per second: 14290\n",
      "MF_BPR: Epoch 424 of 840. Elapsed time 29.88 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 9.33E+00. Sample per second: 13818\n",
      "MF_BPR: Epoch 425 of 840. Elapsed time 29.95 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 9.40E+00. Sample per second: 16562\n",
      "MF_BPR: Epoch 426 of 840. Elapsed time 30.02 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 9.45E+00. Sample per second: 15984\n",
      "MF_BPR: Epoch 427 of 840. Elapsed time 30.09 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 9.46E+00. Sample per second: 15422\n",
      "MF_BPR: Epoch 428 of 840. Elapsed time 30.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 9.53E+00. Sample per second: 14801\n",
      "MF_BPR: Epoch 429 of 840. Elapsed time 30.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 9.51E+00. Sample per second: 14273\n",
      "MF_BPR: Epoch 430 of 840. Elapsed time 30.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 9.50E+00. Sample per second: 13757\n",
      "MF_BPR: Epoch 431 of 840. Elapsed time 30.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 9.57E+00. Sample per second: 16328\n",
      "MF_BPR: Epoch 432 of 840. Elapsed time 30.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 9.66E+00. Sample per second: 15727\n",
      "MF_BPR: Epoch 433 of 840. Elapsed time 30.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 9.62E+00. Sample per second: 15149\n",
      "MF_BPR: Epoch 434 of 840. Elapsed time 30.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 9.72E+00. Sample per second: 14452\n",
      "MF_BPR: Epoch 435 of 840. Elapsed time 30.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 9.75E+00. Sample per second: 13965\n",
      "MF_BPR: Epoch 436 of 840. Elapsed time 30.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.18 seconds. BPR loss 9.74E+00. Sample per second: 16728\n",
      "MF_BPR: Epoch 437 of 840. Elapsed time 30.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 9.73E+00. Sample per second: 16112\n",
      "MF_BPR: Epoch 438 of 840. Elapsed time 30.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 9.69E+00. Sample per second: 15533\n",
      "MF_BPR: Epoch 439 of 840. Elapsed time 30.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 9.79E+00. Sample per second: 14992\n",
      "MF_BPR: Epoch 440 of 840. Elapsed time 31.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 9.80E+00. Sample per second: 14537\n",
      "MF_BPR: Epoch 441 of 840. Elapsed time 31.07 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.98 seconds. BPR loss 9.74E+00. Sample per second: 14032\n",
      "MF_BPR: Epoch 442 of 840. Elapsed time 31.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.17 seconds. BPR loss 9.85E+00. Sample per second: 13526\n",
      "MF_BPR: Epoch 443 of 840. Elapsed time 31.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 9.97E+00. Sample per second: 16153\n",
      "MF_BPR: Epoch 444 of 840. Elapsed time 31.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 9.89E+00. Sample per second: 15549\n",
      "MF_BPR: Epoch 445 of 840. Elapsed time 31.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 9.96E+00. Sample per second: 15021\n",
      "MF_BPR: Epoch 446 of 840. Elapsed time 31.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 9.90E+00. Sample per second: 14581\n",
      "MF_BPR: Epoch 447 of 840. Elapsed time 31.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 9.91E+00. Sample per second: 14174\n",
      "MF_BPR: Epoch 448 of 840. Elapsed time 31.55 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 1.01E+01. Sample per second: 13770\n",
      "MF_BPR: Epoch 449 of 840. Elapsed time 31.62 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 1.00E+01. Sample per second: 16558\n",
      "MF_BPR: Epoch 450 of 840. Elapsed time 31.69 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 1.00E+01. Sample per second: 15929\n",
      "MF_BPR: Epoch 451 of 840. Elapsed time 31.76 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.52 seconds. BPR loss 1.00E+01. Sample per second: 15468\n",
      "MF_BPR: Epoch 452 of 840. Elapsed time 31.83 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 1.01E+01. Sample per second: 15008\n",
      "MF_BPR: Epoch 453 of 840. Elapsed time 31.90 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.02E+01. Sample per second: 14561\n",
      "MF_BPR: Epoch 454 of 840. Elapsed time 31.97 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 1.01E+01. Sample per second: 14149\n",
      "MF_BPR: Epoch 455 of 840. Elapsed time 32.04 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.02E+01. Sample per second: 13647\n",
      "MF_BPR: Epoch 456 of 840. Elapsed time 32.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 1.01E+01. Sample per second: 16183\n",
      "MF_BPR: Epoch 457 of 840. Elapsed time 32.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.02E+01. Sample per second: 15663\n",
      "MF_BPR: Epoch 458 of 840. Elapsed time 32.24 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 1.02E+01. Sample per second: 15191\n",
      "MF_BPR: Epoch 459 of 840. Elapsed time 32.31 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.73 seconds. BPR loss 1.02E+01. Sample per second: 14764\n",
      "MF_BPR: Epoch 460 of 840. Elapsed time 32.38 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 1.02E+01. Sample per second: 14382\n",
      "MF_BPR: Epoch 461 of 840. Elapsed time 32.45 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 1.04E+01. Sample per second: 14017\n",
      "MF_BPR: Epoch 462 of 840. Elapsed time 32.52 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.18 seconds. BPR loss 1.04E+01. Sample per second: 13492\n",
      "MF_BPR: Epoch 463 of 840. Elapsed time 32.59 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.04E+01. Sample per second: 16089\n",
      "MF_BPR: Epoch 464 of 840. Elapsed time 32.66 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.05E+01. Sample per second: 15440\n",
      "MF_BPR: Epoch 465 of 840. Elapsed time 32.73 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 1.05E+01. Sample per second: 14833\n",
      "MF_BPR: Epoch 466 of 840. Elapsed time 32.80 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.04E+01. Sample per second: 14266\n",
      "MF_BPR: Epoch 467 of 840. Elapsed time 32.87 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.04E+01. Sample per second: 13805\n",
      "MF_BPR: Epoch 468 of 840. Elapsed time 32.94 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.20 seconds. BPR loss 1.05E+01. Sample per second: 16650\n",
      "MF_BPR: Epoch 469 of 840. Elapsed time 33.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 1.05E+01. Sample per second: 15919\n",
      "MF_BPR: Epoch 470 of 840. Elapsed time 33.08 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 1.06E+01. Sample per second: 15227\n",
      "MF_BPR: Epoch 471 of 840. Elapsed time 33.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 1.05E+01. Sample per second: 14683\n",
      "MF_BPR: Epoch 472 of 840. Elapsed time 33.22 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.06E+01. Sample per second: 14255\n",
      "MF_BPR: Epoch 473 of 840. Elapsed time 33.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 1.06E+01. Sample per second: 13878\n",
      "MF_BPR: Epoch 474 of 840. Elapsed time 33.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.17 seconds. BPR loss 1.07E+01. Sample per second: 16756\n",
      "MF_BPR: Epoch 475 of 840. Elapsed time 33.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 1.07E+01. Sample per second: 16235\n",
      "MF_BPR: Epoch 476 of 840. Elapsed time 33.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 1.07E+01. Sample per second: 15727\n",
      "MF_BPR: Epoch 477 of 840. Elapsed time 33.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 1.07E+01. Sample per second: 15063\n",
      "MF_BPR: Epoch 478 of 840. Elapsed time 33.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 1.07E+01. Sample per second: 14217\n",
      "MF_BPR: Epoch 479 of 840. Elapsed time 33.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 1.07E+01. Sample per second: 13789\n",
      "MF_BPR: Epoch 480 of 840. Elapsed time 33.77 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 1.08E+01. Sample per second: 16550\n",
      "MF_BPR: Epoch 481 of 840. Elapsed time 33.84 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 1.08E+01. Sample per second: 16002\n",
      "MF_BPR: Epoch 482 of 840. Elapsed time 33.91 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.08E+01. Sample per second: 15431\n",
      "MF_BPR: Epoch 483 of 840. Elapsed time 33.98 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 1.08E+01. Sample per second: 14904\n",
      "MF_BPR: Epoch 484 of 840. Elapsed time 34.05 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 1.08E+01. Sample per second: 14434\n",
      "MF_BPR: Epoch 485 of 840. Elapsed time 34.12 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 1.08E+01. Sample per second: 14011\n",
      "MF_BPR: Epoch 486 of 840. Elapsed time 34.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.30 seconds. BPR loss 1.09E+01. Sample per second: 13179\n",
      "MF_BPR: Epoch 487 of 840. Elapsed time 34.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 1.10E+01. Sample per second: 15579\n",
      "MF_BPR: Epoch 488 of 840. Elapsed time 34.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.09E+01. Sample per second: 15151\n",
      "MF_BPR: Epoch 489 of 840. Elapsed time 34.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 1.08E+01. Sample per second: 14610\n",
      "MF_BPR: Epoch 490 of 840. Elapsed time 34.47 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 1.10E+01. Sample per second: 14191\n",
      "MF_BPR: Epoch 491 of 840. Elapsed time 34.54 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.09E+01. Sample per second: 13810\n",
      "MF_BPR: Epoch 492 of 840. Elapsed time 34.60 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.20 seconds. BPR loss 1.11E+01. Sample per second: 16633\n",
      "MF_BPR: Epoch 493 of 840. Elapsed time 34.67 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.11E+01. Sample per second: 16086\n",
      "MF_BPR: Epoch 494 of 840. Elapsed time 34.74 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.11E+01. Sample per second: 15528\n",
      "MF_BPR: Epoch 495 of 840. Elapsed time 34.81 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 1.11E+01. Sample per second: 14949\n",
      "MF_BPR: Epoch 496 of 840. Elapsed time 34.88 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.12E+01. Sample per second: 14416\n",
      "MF_BPR: Epoch 497 of 840. Elapsed time 34.95 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 1.11E+01. Sample per second: 14017\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 498 of 840. Elapsed time 35.02 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.12E+01. Sample per second: 13636\n",
      "MF_BPR: Epoch 499 of 840. Elapsed time 35.09 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.26 seconds. BPR loss 1.12E+01. Sample per second: 16421\n",
      "MF_BPR: Epoch 500 of 840. Elapsed time 35.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 1.12E+01. Sample per second: 15119\n",
      "MF_BPR: Epoch 501 of 840. Elapsed time 35.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.13E+01. Sample per second: 13824\n",
      "MF_BPR: Epoch 502 of 840. Elapsed time 35.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.19 seconds. BPR loss 1.12E+01. Sample per second: 16665\n",
      "MF_BPR: Epoch 503 of 840. Elapsed time 35.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 1.12E+01. Sample per second: 16134\n",
      "MF_BPR: Epoch 504 of 840. Elapsed time 35.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 1.13E+01. Sample per second: 15692\n",
      "MF_BPR: Epoch 505 of 840. Elapsed time 35.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.58 seconds. BPR loss 1.12E+01. Sample per second: 15243\n",
      "MF_BPR: Epoch 506 of 840. Elapsed time 35.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 1.13E+01. Sample per second: 14833\n",
      "MF_BPR: Epoch 507 of 840. Elapsed time 35.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.14E+01. Sample per second: 14409\n",
      "MF_BPR: Epoch 508 of 840. Elapsed time 35.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.98 seconds. BPR loss 1.15E+01. Sample per second: 14021\n",
      "MF_BPR: Epoch 509 of 840. Elapsed time 35.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.15E+01. Sample per second: 13659\n",
      "MF_BPR: Epoch 510 of 840. Elapsed time 35.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.15E+01. Sample per second: 16334\n",
      "MF_BPR: Epoch 511 of 840. Elapsed time 35.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 1.15E+01. Sample per second: 15712\n",
      "MF_BPR: Epoch 512 of 840. Elapsed time 35.99 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.15E+01. Sample per second: 15157\n",
      "MF_BPR: Epoch 513 of 840. Elapsed time 36.06 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 1.15E+01. Sample per second: 14598\n",
      "MF_BPR: Epoch 514 of 840. Elapsed time 36.13 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 1.16E+01. Sample per second: 14205\n",
      "MF_BPR: Epoch 515 of 840. Elapsed time 36.20 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.16E+01. Sample per second: 13845\n",
      "MF_BPR: Epoch 516 of 840. Elapsed time 36.27 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.16 seconds. BPR loss 1.15E+01. Sample per second: 16787\n",
      "MF_BPR: Epoch 517 of 840. Elapsed time 36.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 1.16E+01. Sample per second: 16298\n",
      "MF_BPR: Epoch 518 of 840. Elapsed time 36.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 1.15E+01. Sample per second: 15826\n",
      "MF_BPR: Epoch 519 of 840. Elapsed time 36.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 1.16E+01. Sample per second: 15327\n",
      "MF_BPR: Epoch 520 of 840. Elapsed time 36.55 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 1.16E+01. Sample per second: 14841\n",
      "MF_BPR: Epoch 521 of 840. Elapsed time 36.61 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 1.17E+01. Sample per second: 14316\n",
      "MF_BPR: Epoch 522 of 840. Elapsed time 36.68 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.17E+01. Sample per second: 13831\n",
      "MF_BPR: Epoch 523 of 840. Elapsed time 36.75 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.25 seconds. BPR loss 1.17E+01. Sample per second: 16448\n",
      "MF_BPR: Epoch 524 of 840. Elapsed time 36.82 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 1.17E+01. Sample per second: 15713\n",
      "MF_BPR: Epoch 525 of 840. Elapsed time 36.89 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 1.18E+01. Sample per second: 15075\n",
      "MF_BPR: Epoch 526 of 840. Elapsed time 36.96 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 1.18E+01. Sample per second: 14596\n",
      "MF_BPR: Epoch 527 of 840. Elapsed time 37.03 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 1.17E+01. Sample per second: 14173\n",
      "MF_BPR: Epoch 528 of 840. Elapsed time 37.10 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 1.19E+01. Sample per second: 13776\n",
      "MF_BPR: Epoch 529 of 840. Elapsed time 37.17 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 1.17E+01. Sample per second: 16493\n",
      "MF_BPR: Epoch 530 of 840. Elapsed time 37.24 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 1.18E+01. Sample per second: 15886\n",
      "MF_BPR: Epoch 531 of 840. Elapsed time 37.31 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.18E+01. Sample per second: 15363\n",
      "MF_BPR: Epoch 532 of 840. Elapsed time 37.38 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 1.19E+01. Sample per second: 14918\n",
      "MF_BPR: Epoch 533 of 840. Elapsed time 37.45 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 1.19E+01. Sample per second: 14497\n",
      "MF_BPR: Epoch 534 of 840. Elapsed time 37.52 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 1.20E+01. Sample per second: 13966\n",
      "MF_BPR: Epoch 535 of 840. Elapsed time 37.59 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.14 seconds. BPR loss 1.20E+01. Sample per second: 16861\n",
      "MF_BPR: Epoch 536 of 840. Elapsed time 37.66 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 1.20E+01. Sample per second: 16175\n",
      "MF_BPR: Epoch 537 of 840. Elapsed time 37.73 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.20E+01. Sample per second: 15527\n",
      "MF_BPR: Epoch 538 of 840. Elapsed time 37.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 1.20E+01. Sample per second: 14978\n",
      "MF_BPR: Epoch 539 of 840. Elapsed time 37.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.20E+01. Sample per second: 14416\n",
      "MF_BPR: Epoch 540 of 840. Elapsed time 37.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.20E+01. Sample per second: 13890\n",
      "MF_BPR: Epoch 541 of 840. Elapsed time 38.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 1.21E+01. Sample per second: 16581\n",
      "MF_BPR: Epoch 542 of 840. Elapsed time 38.07 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 1.21E+01. Sample per second: 15915\n",
      "MF_BPR: Epoch 543 of 840. Elapsed time 38.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.58 seconds. BPR loss 1.21E+01. Sample per second: 15259\n",
      "MF_BPR: Epoch 544 of 840. Elapsed time 38.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.22E+01. Sample per second: 14740\n",
      "MF_BPR: Epoch 545 of 840. Elapsed time 38.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.22E+01. Sample per second: 14271\n",
      "MF_BPR: Epoch 546 of 840. Elapsed time 38.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.22E+01. Sample per second: 13842\n",
      "MF_BPR: Epoch 547 of 840. Elapsed time 38.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.23 seconds. BPR loss 1.22E+01. Sample per second: 16535\n",
      "MF_BPR: Epoch 548 of 840. Elapsed time 38.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 1.22E+01. Sample per second: 15945\n",
      "MF_BPR: Epoch 549 of 840. Elapsed time 38.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.21E+01. Sample per second: 15420\n",
      "MF_BPR: Epoch 550 of 840. Elapsed time 38.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 1.22E+01. Sample per second: 14707\n",
      "MF_BPR: Epoch 551 of 840. Elapsed time 38.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.23E+01. Sample per second: 14257\n",
      "MF_BPR: Epoch 552 of 840. Elapsed time 38.77 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.22E+01. Sample per second: 13806\n",
      "MF_BPR: Epoch 553 of 840. Elapsed time 38.84 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.23 seconds. BPR loss 1.23E+01. Sample per second: 16503\n",
      "MF_BPR: Epoch 554 of 840. Elapsed time 38.91 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 1.24E+01. Sample per second: 15858\n",
      "MF_BPR: Epoch 555 of 840. Elapsed time 38.98 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 1.23E+01. Sample per second: 15312\n",
      "MF_BPR: Epoch 556 of 840. Elapsed time 39.05 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.23E+01. Sample per second: 14855\n",
      "MF_BPR: Epoch 557 of 840. Elapsed time 39.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 1.23E+01. Sample per second: 14202\n",
      "MF_BPR: Epoch 558 of 840. Elapsed time 39.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 1.25E+01. Sample per second: 13765\n",
      "MF_BPR: Epoch 559 of 840. Elapsed time 39.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 1.24E+01. Sample per second: 16212\n",
      "MF_BPR: Epoch 560 of 840. Elapsed time 39.32 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 1.25E+01. Sample per second: 14610\n",
      "MF_BPR: Epoch 561 of 840. Elapsed time 39.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.76 seconds. BPR loss 1.25E+01. Sample per second: 12132\n",
      "MF_BPR: Epoch 562 of 840. Elapsed time 39.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.25E+01. Sample per second: 13819\n",
      "MF_BPR: Epoch 563 of 840. Elapsed time 39.55 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 1.25E+01. Sample per second: 16072\n",
      "MF_BPR: Epoch 564 of 840. Elapsed time 39.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.25E+01. Sample per second: 15145\n",
      "MF_BPR: Epoch 565 of 840. Elapsed time 39.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 1.25E+01. Sample per second: 14426\n",
      "MF_BPR: Epoch 566 of 840. Elapsed time 39.77 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 1.26E+01. Sample per second: 13773\n",
      "MF_BPR: Epoch 567 of 840. Elapsed time 39.84 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 1.26E+01. Sample per second: 16287\n",
      "MF_BPR: Epoch 568 of 840. Elapsed time 39.91 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.26E+01. Sample per second: 15442\n",
      "MF_BPR: Epoch 569 of 840. Elapsed time 39.98 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.26E+01. Sample per second: 14875\n",
      "MF_BPR: Epoch 570 of 840. Elapsed time 40.05 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.26E+01. Sample per second: 14268\n",
      "MF_BPR: Epoch 571 of 840. Elapsed time 40.12 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.27E+01. Sample per second: 13661\n",
      "MF_BPR: Epoch 572 of 840. Elapsed time 40.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.32 seconds. BPR loss 1.27E+01. Sample per second: 16178\n",
      "MF_BPR: Epoch 573 of 840. Elapsed time 40.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 1.26E+01. Sample per second: 15503\n",
      "MF_BPR: Epoch 574 of 840. Elapsed time 40.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 1.27E+01. Sample per second: 14928\n",
      "MF_BPR: Epoch 575 of 840. Elapsed time 40.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.89 seconds. BPR loss 1.28E+01. Sample per second: 14286\n",
      "MF_BPR: Epoch 576 of 840. Elapsed time 40.47 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.28E+01. Sample per second: 13810\n",
      "MF_BPR: Epoch 577 of 840. Elapsed time 40.54 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.23 seconds. BPR loss 1.28E+01. Sample per second: 16522\n",
      "MF_BPR: Epoch 578 of 840. Elapsed time 40.61 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 1.28E+01. Sample per second: 15814\n",
      "MF_BPR: Epoch 579 of 840. Elapsed time 40.68 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 1.28E+01. Sample per second: 15301\n",
      "MF_BPR: Epoch 580 of 840. Elapsed time 40.75 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 1.28E+01. Sample per second: 14494\n",
      "MF_BPR: Epoch 581 of 840. Elapsed time 40.82 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.29E+01. Sample per second: 13888\n",
      "MF_BPR: Epoch 582 of 840. Elapsed time 40.89 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.27 seconds. BPR loss 1.28E+01. Sample per second: 16363\n",
      "MF_BPR: Epoch 583 of 840. Elapsed time 40.96 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 1.29E+01. Sample per second: 14923\n",
      "MF_BPR: Epoch 584 of 840. Elapsed time 41.03 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.28E+01. Sample per second: 13713\n",
      "MF_BPR: Epoch 585 of 840. Elapsed time 41.10 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 1.28E+01. Sample per second: 16233\n",
      "MF_BPR: Epoch 586 of 840. Elapsed time 41.17 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 1.30E+01. Sample per second: 15375\n",
      "MF_BPR: Epoch 587 of 840. Elapsed time 41.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 1.30E+01. Sample per second: 14617\n",
      "MF_BPR: Epoch 588 of 840. Elapsed time 41.32 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 1.30E+01. Sample per second: 14010\n",
      "MF_BPR: Epoch 589 of 840. Elapsed time 41.39 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.18 seconds. BPR loss 1.30E+01. Sample per second: 13488\n",
      "MF_BPR: Epoch 590 of 840. Elapsed time 41.46 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 1.30E+01. Sample per second: 16024\n",
      "MF_BPR: Epoch 591 of 840. Elapsed time 41.53 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 1.30E+01. Sample per second: 15401\n",
      "MF_BPR: Epoch 592 of 840. Elapsed time 41.60 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.72 seconds. BPR loss 1.30E+01. Sample per second: 14808\n",
      "MF_BPR: Epoch 593 of 840. Elapsed time 41.67 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 1.30E+01. Sample per second: 14195\n",
      "MF_BPR: Epoch 594 of 840. Elapsed time 41.74 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.20 seconds. BPR loss 1.30E+01. Sample per second: 13438\n",
      "MF_BPR: Epoch 595 of 840. Elapsed time 41.81 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 1.32E+01. Sample per second: 15750\n",
      "MF_BPR: Epoch 596 of 840. Elapsed time 41.88 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 1.31E+01. Sample per second: 14939\n",
      "MF_BPR: Epoch 597 of 840. Elapsed time 41.95 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.96 seconds. BPR loss 1.32E+01. Sample per second: 14097\n",
      "MF_BPR: Epoch 598 of 840. Elapsed time 42.02 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.17 seconds. BPR loss 1.31E+01. Sample per second: 13504\n",
      "MF_BPR: Epoch 599 of 840. Elapsed time 42.09 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 1.32E+01. Sample per second: 16016\n",
      "MF_BPR: Epoch 600 of 840. Elapsed time 42.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.31E+01. Sample per second: 15172\n",
      "MF_BPR: Epoch 601 of 840. Elapsed time 42.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.31E+01. Sample per second: 14555\n",
      "MF_BPR: Epoch 602 of 840. Elapsed time 42.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.96 seconds. BPR loss 1.32E+01. Sample per second: 14097\n",
      "MF_BPR: Epoch 603 of 840. Elapsed time 42.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.14 seconds. BPR loss 1.32E+01. Sample per second: 13583\n",
      "MF_BPR: Epoch 604 of 840. Elapsed time 42.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.36 seconds. BPR loss 1.32E+01. Sample per second: 16018\n",
      "MF_BPR: Epoch 605 of 840. Elapsed time 42.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.33E+01. Sample per second: 15343\n",
      "MF_BPR: Epoch 606 of 840. Elapsed time 42.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.33E+01. Sample per second: 14568\n",
      "MF_BPR: Epoch 607 of 840. Elapsed time 42.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.33E+01. Sample per second: 13712\n",
      "MF_BPR: Epoch 608 of 840. Elapsed time 42.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.33E+01. Sample per second: 15675\n",
      "MF_BPR: Epoch 609 of 840. Elapsed time 42.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.33E+01. Sample per second: 14878\n",
      "MF_BPR: Epoch 610 of 840. Elapsed time 42.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 1.32E+01. Sample per second: 14331\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 611 of 840. Elapsed time 42.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 1.34E+01. Sample per second: 13857\n",
      "MF_BPR: Epoch 612 of 840. Elapsed time 43.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 1.34E+01. Sample per second: 16062\n",
      "MF_BPR: Epoch 613 of 840. Elapsed time 43.08 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.57 seconds. BPR loss 1.34E+01. Sample per second: 15298\n",
      "MF_BPR: Epoch 614 of 840. Elapsed time 43.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 1.34E+01. Sample per second: 14577\n",
      "MF_BPR: Epoch 615 of 840. Elapsed time 43.22 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.35E+01. Sample per second: 13889\n",
      "MF_BPR: Epoch 616 of 840. Elapsed time 43.29 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.27 seconds. BPR loss 1.33E+01. Sample per second: 16367\n",
      "MF_BPR: Epoch 617 of 840. Elapsed time 43.36 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 1.35E+01. Sample per second: 15178\n",
      "MF_BPR: Epoch 618 of 840. Elapsed time 43.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 1.35E+01. Sample per second: 14507\n",
      "MF_BPR: Epoch 619 of 840. Elapsed time 43.50 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.34E+01. Sample per second: 13818\n",
      "MF_BPR: Epoch 620 of 840. Elapsed time 43.57 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.35E+01. Sample per second: 16327\n",
      "MF_BPR: Epoch 621 of 840. Elapsed time 43.64 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 1.35E+01. Sample per second: 15613\n",
      "MF_BPR: Epoch 622 of 840. Elapsed time 43.71 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 1.35E+01. Sample per second: 14826\n",
      "MF_BPR: Epoch 623 of 840. Elapsed time 43.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 1.36E+01. Sample per second: 14154\n",
      "MF_BPR: Epoch 624 of 840. Elapsed time 43.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.19 seconds. BPR loss 1.36E+01. Sample per second: 13462\n",
      "MF_BPR: Epoch 625 of 840. Elapsed time 43.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.35E+01. Sample per second: 15433\n",
      "MF_BPR: Epoch 626 of 840. Elapsed time 44.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 1.36E+01. Sample per second: 13559\n",
      "MF_BPR: Epoch 627 of 840. Elapsed time 44.07 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 1.37E+01. Sample per second: 16128\n",
      "MF_BPR: Epoch 628 of 840. Elapsed time 44.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.51 seconds. BPR loss 1.37E+01. Sample per second: 15511\n",
      "MF_BPR: Epoch 629 of 840. Elapsed time 44.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 1.37E+01. Sample per second: 14847\n",
      "MF_BPR: Epoch 630 of 840. Elapsed time 44.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.97 seconds. BPR loss 1.37E+01. Sample per second: 14068\n",
      "MF_BPR: Epoch 631 of 840. Elapsed time 44.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.13 seconds. BPR loss 1.37E+01. Sample per second: 13629\n",
      "MF_BPR: Epoch 632 of 840. Elapsed time 44.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 1.37E+01. Sample per second: 16002\n",
      "MF_BPR: Epoch 633 of 840. Elapsed time 44.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 1.38E+01. Sample per second: 15185\n",
      "MF_BPR: Epoch 634 of 840. Elapsed time 44.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.38E+01. Sample per second: 14229\n",
      "MF_BPR: Epoch 635 of 840. Elapsed time 44.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.21 seconds. BPR loss 1.38E+01. Sample per second: 13403\n",
      "MF_BPR: Epoch 636 of 840. Elapsed time 44.71 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 1.37E+01. Sample per second: 15901\n",
      "MF_BPR: Epoch 637 of 840. Elapsed time 44.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.62 seconds. BPR loss 1.37E+01. Sample per second: 15137\n",
      "MF_BPR: Epoch 638 of 840. Elapsed time 44.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.38E+01. Sample per second: 14455\n",
      "MF_BPR: Epoch 639 of 840. Elapsed time 44.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.39E+01. Sample per second: 13822\n",
      "MF_BPR: Epoch 640 of 840. Elapsed time 44.99 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 1.39E+01. Sample per second: 16611\n",
      "MF_BPR: Epoch 641 of 840. Elapsed time 45.06 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.38E+01. Sample per second: 16102\n",
      "MF_BPR: Epoch 642 of 840. Elapsed time 45.13 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.52 seconds. BPR loss 1.39E+01. Sample per second: 15461\n",
      "MF_BPR: Epoch 643 of 840. Elapsed time 45.20 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 1.38E+01. Sample per second: 14904\n",
      "MF_BPR: Epoch 644 of 840. Elapsed time 45.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.39E+01. Sample per second: 14232\n",
      "MF_BPR: Epoch 645 of 840. Elapsed time 45.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.39E+01. Sample per second: 13689\n",
      "MF_BPR: Epoch 646 of 840. Elapsed time 45.40 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.27 seconds. BPR loss 1.39E+01. Sample per second: 16366\n",
      "MF_BPR: Epoch 647 of 840. Elapsed time 45.47 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.44 seconds. BPR loss 1.39E+01. Sample per second: 15725\n",
      "MF_BPR: Epoch 648 of 840. Elapsed time 45.54 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.40E+01. Sample per second: 15154\n",
      "MF_BPR: Epoch 649 of 840. Elapsed time 45.61 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 1.40E+01. Sample per second: 14613\n",
      "MF_BPR: Epoch 650 of 840. Elapsed time 45.68 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 1.40E+01. Sample per second: 14157\n",
      "MF_BPR: Epoch 651 of 840. Elapsed time 45.75 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 1.40E+01. Sample per second: 13737\n",
      "MF_BPR: Epoch 652 of 840. Elapsed time 45.82 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.26 seconds. BPR loss 1.40E+01. Sample per second: 16397\n",
      "MF_BPR: Epoch 653 of 840. Elapsed time 45.89 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 1.40E+01. Sample per second: 15883\n",
      "MF_BPR: Epoch 654 of 840. Elapsed time 45.96 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.41E+01. Sample per second: 15350\n",
      "MF_BPR: Epoch 655 of 840. Elapsed time 46.03 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 1.41E+01. Sample per second: 14894\n",
      "MF_BPR: Epoch 656 of 840. Elapsed time 46.10 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.41E+01. Sample per second: 14456\n",
      "MF_BPR: Epoch 657 of 840. Elapsed time 46.17 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 1.40E+01. Sample per second: 13964\n",
      "MF_BPR: Epoch 658 of 840. Elapsed time 46.24 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.15 seconds. BPR loss 1.41E+01. Sample per second: 16843\n",
      "MF_BPR: Epoch 659 of 840. Elapsed time 46.31 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 1.42E+01. Sample per second: 16231\n",
      "MF_BPR: Epoch 660 of 840. Elapsed time 46.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.41E+01. Sample per second: 15671\n",
      "MF_BPR: Epoch 661 of 840. Elapsed time 46.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.41E+01. Sample per second: 15157\n",
      "MF_BPR: Epoch 662 of 840. Elapsed time 46.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 1.42E+01. Sample per second: 14653\n",
      "MF_BPR: Epoch 663 of 840. Elapsed time 46.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 1.42E+01. Sample per second: 14123\n",
      "MF_BPR: Epoch 664 of 840. Elapsed time 46.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.41E+01. Sample per second: 13637\n",
      "MF_BPR: Epoch 665 of 840. Elapsed time 46.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.31 seconds. BPR loss 1.42E+01. Sample per second: 16226\n",
      "MF_BPR: Epoch 666 of 840. Elapsed time 46.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 1.43E+01. Sample per second: 15594\n",
      "MF_BPR: Epoch 667 of 840. Elapsed time 46.86 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 1.42E+01. Sample per second: 15056\n",
      "MF_BPR: Epoch 668 of 840. Elapsed time 46.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 1.42E+01. Sample per second: 14613\n",
      "MF_BPR: Epoch 669 of 840. Elapsed time 47.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.94 seconds. BPR loss 1.42E+01. Sample per second: 14156\n",
      "MF_BPR: Epoch 670 of 840. Elapsed time 47.07 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 1.43E+01. Sample per second: 13769\n",
      "MF_BPR: Epoch 671 of 840. Elapsed time 47.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 1.43E+01. Sample per second: 16589\n",
      "MF_BPR: Epoch 672 of 840. Elapsed time 47.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.35 seconds. BPR loss 1.43E+01. Sample per second: 16081\n",
      "MF_BPR: Epoch 673 of 840. Elapsed time 47.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 1.43E+01. Sample per second: 15595\n",
      "MF_BPR: Epoch 674 of 840. Elapsed time 47.34 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.43E+01. Sample per second: 15149\n",
      "MF_BPR: Epoch 675 of 840. Elapsed time 47.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 1.44E+01. Sample per second: 14683\n",
      "MF_BPR: Epoch 676 of 840. Elapsed time 47.48 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.44E+01. Sample per second: 14244\n",
      "MF_BPR: Epoch 677 of 840. Elapsed time 47.55 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.44E+01. Sample per second: 13704\n",
      "MF_BPR: Epoch 678 of 840. Elapsed time 47.62 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 1.44E+01. Sample per second: 16148\n",
      "MF_BPR: Epoch 679 of 840. Elapsed time 47.69 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.52 seconds. BPR loss 1.44E+01. Sample per second: 15470\n",
      "MF_BPR: Epoch 680 of 840. Elapsed time 47.76 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 1.44E+01. Sample per second: 14847\n",
      "MF_BPR: Epoch 681 of 840. Elapsed time 47.83 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.45E+01. Sample per second: 14259\n",
      "MF_BPR: Epoch 682 of 840. Elapsed time 47.90 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 1.45E+01. Sample per second: 13731\n",
      "MF_BPR: Epoch 683 of 840. Elapsed time 47.97 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.45E+01. Sample per second: 16315\n",
      "MF_BPR: Epoch 684 of 840. Elapsed time 48.04 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 1.45E+01. Sample per second: 15716\n",
      "MF_BPR: Epoch 685 of 840. Elapsed time 48.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 1.45E+01. Sample per second: 15221\n",
      "MF_BPR: Epoch 686 of 840. Elapsed time 48.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.77 seconds. BPR loss 1.45E+01. Sample per second: 14665\n",
      "MF_BPR: Epoch 687 of 840. Elapsed time 48.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.92 seconds. BPR loss 1.46E+01. Sample per second: 14217\n",
      "MF_BPR: Epoch 688 of 840. Elapsed time 48.32 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.45E+01. Sample per second: 13831\n",
      "MF_BPR: Epoch 689 of 840. Elapsed time 48.39 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.20 seconds. BPR loss 1.47E+01. Sample per second: 16641\n",
      "MF_BPR: Epoch 690 of 840. Elapsed time 48.46 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 1.47E+01. Sample per second: 15951\n",
      "MF_BPR: Epoch 691 of 840. Elapsed time 48.53 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.46E+01. Sample per second: 15412\n",
      "MF_BPR: Epoch 692 of 840. Elapsed time 48.60 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.47E+01. Sample per second: 14858\n",
      "MF_BPR: Epoch 693 of 840. Elapsed time 48.66 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 1.46E+01. Sample per second: 14351\n",
      "MF_BPR: Epoch 694 of 840. Elapsed time 48.73 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 1.47E+01. Sample per second: 13945\n",
      "MF_BPR: Epoch 695 of 840. Elapsed time 48.80 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.16 seconds. BPR loss 1.47E+01. Sample per second: 16814\n",
      "MF_BPR: Epoch 696 of 840. Elapsed time 48.87 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 1.47E+01. Sample per second: 16261\n",
      "MF_BPR: Epoch 697 of 840. Elapsed time 48.94 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.45 seconds. BPR loss 1.47E+01. Sample per second: 15713\n",
      "MF_BPR: Epoch 698 of 840. Elapsed time 49.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.60 seconds. BPR loss 1.48E+01. Sample per second: 15207\n",
      "MF_BPR: Epoch 699 of 840. Elapsed time 49.08 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.79 seconds. BPR loss 1.48E+01. Sample per second: 14600\n",
      "MF_BPR: Epoch 700 of 840. Elapsed time 49.15 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.99 seconds. BPR loss 1.47E+01. Sample per second: 14008\n",
      "MF_BPR: Epoch 701 of 840. Elapsed time 49.22 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.18 seconds. BPR loss 1.48E+01. Sample per second: 13500\n",
      "MF_BPR: Epoch 702 of 840. Elapsed time 49.29 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.48E+01. Sample per second: 16108\n",
      "MF_BPR: Epoch 703 of 840. Elapsed time 49.36 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.48E+01. Sample per second: 15439\n",
      "MF_BPR: Epoch 704 of 840. Elapsed time 49.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.48E+01. Sample per second: 14874\n",
      "MF_BPR: Epoch 705 of 840. Elapsed time 49.50 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 1.48E+01. Sample per second: 14307\n",
      "MF_BPR: Epoch 706 of 840. Elapsed time 49.57 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 1.49E+01. Sample per second: 13873\n",
      "MF_BPR: Epoch 707 of 840. Elapsed time 49.64 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 1.49E+01. Sample per second: 16609\n",
      "MF_BPR: Epoch 708 of 840. Elapsed time 49.71 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.37 seconds. BPR loss 1.49E+01. Sample per second: 15993\n",
      "MF_BPR: Epoch 709 of 840. Elapsed time 49.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.50E+01. Sample per second: 15430\n",
      "MF_BPR: Epoch 710 of 840. Elapsed time 49.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 1.49E+01. Sample per second: 14915\n",
      "MF_BPR: Epoch 711 of 840. Elapsed time 49.91 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.49E+01. Sample per second: 14465\n",
      "MF_BPR: Epoch 712 of 840. Elapsed time 49.98 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.98 seconds. BPR loss 1.50E+01. Sample per second: 14026\n",
      "MF_BPR: Epoch 713 of 840. Elapsed time 50.05 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 1.50E+01. Sample per second: 13581\n",
      "MF_BPR: Epoch 714 of 840. Elapsed time 50.12 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 1.50E+01. Sample per second: 16244\n",
      "MF_BPR: Epoch 715 of 840. Elapsed time 50.19 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.50E+01. Sample per second: 15654\n",
      "MF_BPR: Epoch 716 of 840. Elapsed time 50.26 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.51E+01. Sample per second: 14863\n",
      "MF_BPR: Epoch 717 of 840. Elapsed time 50.33 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.49E+01. Sample per second: 13658\n",
      "MF_BPR: Epoch 718 of 840. Elapsed time 50.41 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.27 seconds. BPR loss 1.51E+01. Sample per second: 16374\n",
      "MF_BPR: Epoch 719 of 840. Elapsed time 50.47 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 1.50E+01. Sample per second: 15840\n",
      "MF_BPR: Epoch 720 of 840. Elapsed time 50.54 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.56 seconds. BPR loss 1.51E+01. Sample per second: 15337\n",
      "MF_BPR: Epoch 721 of 840. Elapsed time 50.61 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.71 seconds. BPR loss 1.52E+01. Sample per second: 14845\n",
      "MF_BPR: Epoch 722 of 840. Elapsed time 50.68 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.87 seconds. BPR loss 1.51E+01. Sample per second: 14352\n",
      "MF_BPR: Epoch 723 of 840. Elapsed time 50.75 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.50E+01. Sample per second: 13900\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 724 of 840. Elapsed time 50.82 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.16 seconds. BPR loss 1.52E+01. Sample per second: 16813\n",
      "MF_BPR: Epoch 725 of 840. Elapsed time 50.89 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 1.51E+01. Sample per second: 16239\n",
      "MF_BPR: Epoch 726 of 840. Elapsed time 50.96 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.51E+01. Sample per second: 15684\n",
      "MF_BPR: Epoch 727 of 840. Elapsed time 51.03 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.65 seconds. BPR loss 1.53E+01. Sample per second: 15022\n",
      "MF_BPR: Epoch 728 of 840. Elapsed time 51.10 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 1.50E+01. Sample per second: 14496\n",
      "MF_BPR: Epoch 729 of 840. Elapsed time 51.17 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 1.52E+01. Sample per second: 13938\n",
      "MF_BPR: Epoch 730 of 840. Elapsed time 51.24 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.24 seconds. BPR loss 1.53E+01. Sample per second: 16468\n",
      "MF_BPR: Epoch 731 of 840. Elapsed time 51.31 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.58 seconds. BPR loss 1.53E+01. Sample per second: 15272\n",
      "MF_BPR: Epoch 732 of 840. Elapsed time 51.38 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.83 seconds. BPR loss 1.52E+01. Sample per second: 14479\n",
      "MF_BPR: Epoch 733 of 840. Elapsed time 51.45 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.53E+01. Sample per second: 13820\n",
      "MF_BPR: Epoch 734 of 840. Elapsed time 51.52 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.21 seconds. BPR loss 1.52E+01. Sample per second: 16580\n",
      "MF_BPR: Epoch 735 of 840. Elapsed time 51.59 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 1.53E+01. Sample per second: 15970\n",
      "MF_BPR: Epoch 736 of 840. Elapsed time 51.66 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 1.53E+01. Sample per second: 15220\n",
      "MF_BPR: Epoch 737 of 840. Elapsed time 51.73 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 1.53E+01. Sample per second: 14678\n",
      "MF_BPR: Epoch 738 of 840. Elapsed time 51.80 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.93 seconds. BPR loss 1.54E+01. Sample per second: 14170\n",
      "MF_BPR: Epoch 739 of 840. Elapsed time 51.87 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.13 seconds. BPR loss 1.53E+01. Sample per second: 13609\n",
      "MF_BPR: Epoch 740 of 840. Elapsed time 51.94 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.41 seconds. BPR loss 1.53E+01. Sample per second: 15863\n",
      "MF_BPR: Epoch 741 of 840. Elapsed time 52.01 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.01 seconds. BPR loss 1.55E+01. Sample per second: 13940\n",
      "MF_BPR: Epoch 742 of 840. Elapsed time 52.09 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.42 seconds. BPR loss 1.53E+01. Sample per second: 15804\n",
      "MF_BPR: Epoch 743 of 840. Elapsed time 52.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 1.55E+01. Sample per second: 14392\n",
      "MF_BPR: Epoch 744 of 840. Elapsed time 52.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.28 seconds. BPR loss 1.55E+01. Sample per second: 13246\n",
      "MF_BPR: Epoch 745 of 840. Elapsed time 52.31 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.54E+01. Sample per second: 14741\n",
      "MF_BPR: Epoch 746 of 840. Elapsed time 52.38 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.34 seconds. BPR loss 1.54E+01. Sample per second: 13095\n",
      "MF_BPR: Epoch 747 of 840. Elapsed time 52.46 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.76 seconds. BPR loss 1.55E+01. Sample per second: 14688\n",
      "MF_BPR: Epoch 748 of 840. Elapsed time 52.53 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.54E+01. Sample per second: 13704\n",
      "MF_BPR: Epoch 749 of 840. Elapsed time 52.60 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.54E+01. Sample per second: 16312\n",
      "MF_BPR: Epoch 750 of 840. Elapsed time 52.67 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 1.55E+01. Sample per second: 15637\n",
      "MF_BPR: Epoch 751 of 840. Elapsed time 52.74 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 1.56E+01. Sample per second: 15050\n",
      "MF_BPR: Epoch 752 of 840. Elapsed time 52.81 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.55E+01. Sample per second: 14398\n",
      "MF_BPR: Epoch 753 of 840. Elapsed time 52.88 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 1.55E+01. Sample per second: 13778\n",
      "MF_BPR: Epoch 754 of 840. Elapsed time 52.95 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 1.56E+01. Sample per second: 16305\n",
      "MF_BPR: Epoch 755 of 840. Elapsed time 53.02 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.47 seconds. BPR loss 1.56E+01. Sample per second: 15639\n",
      "MF_BPR: Epoch 756 of 840. Elapsed time 53.09 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.66 seconds. BPR loss 1.56E+01. Sample per second: 14985\n",
      "MF_BPR: Epoch 757 of 840. Elapsed time 53.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.56E+01. Sample per second: 14229\n",
      "MF_BPR: Epoch 758 of 840. Elapsed time 53.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.56E+01. Sample per second: 13693\n",
      "MF_BPR: Epoch 759 of 840. Elapsed time 53.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.33 seconds. BPR loss 1.57E+01. Sample per second: 16141\n",
      "MF_BPR: Epoch 760 of 840. Elapsed time 53.38 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.58E+01. Sample per second: 15539\n",
      "MF_BPR: Epoch 761 of 840. Elapsed time 53.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.67 seconds. BPR loss 1.57E+01. Sample per second: 14958\n",
      "MF_BPR: Epoch 762 of 840. Elapsed time 53.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.86 seconds. BPR loss 1.57E+01. Sample per second: 14377\n",
      "MF_BPR: Epoch 763 of 840. Elapsed time 53.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.06 seconds. BPR loss 1.57E+01. Sample per second: 13819\n",
      "MF_BPR: Epoch 764 of 840. Elapsed time 53.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.27 seconds. BPR loss 1.58E+01. Sample per second: 16370\n",
      "MF_BPR: Epoch 765 of 840. Elapsed time 53.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.59E+01. Sample per second: 15671\n",
      "MF_BPR: Epoch 766 of 840. Elapsed time 53.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.64 seconds. BPR loss 1.58E+01. Sample per second: 15069\n",
      "MF_BPR: Epoch 767 of 840. Elapsed time 53.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.84 seconds. BPR loss 1.58E+01. Sample per second: 14432\n",
      "MF_BPR: Epoch 768 of 840. Elapsed time 53.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.58E+01. Sample per second: 13886\n",
      "MF_BPR: Epoch 769 of 840. Elapsed time 54.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.23 seconds. BPR loss 1.58E+01. Sample per second: 16537\n",
      "MF_BPR: Epoch 770 of 840. Elapsed time 54.07 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.38 seconds. BPR loss 1.59E+01. Sample per second: 15936\n",
      "MF_BPR: Epoch 771 of 840. Elapsed time 54.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.58E+01. Sample per second: 15168\n",
      "MF_BPR: Epoch 772 of 840. Elapsed time 54.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.78 seconds. BPR loss 1.58E+01. Sample per second: 14632\n",
      "MF_BPR: Epoch 773 of 840. Elapsed time 54.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 1.59E+01. Sample per second: 14128\n",
      "MF_BPR: Epoch 774 of 840. Elapsed time 54.35 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.12 seconds. BPR loss 1.60E+01. Sample per second: 13657\n",
      "MF_BPR: Epoch 775 of 840. Elapsed time 54.42 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 1.57E+01. Sample per second: 16264\n",
      "MF_BPR: Epoch 776 of 840. Elapsed time 54.49 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.58E+01. Sample per second: 15676\n",
      "MF_BPR: Epoch 777 of 840. Elapsed time 54.56 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.60E+01. Sample per second: 15172\n",
      "MF_BPR: Epoch 778 of 840. Elapsed time 54.63 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.59E+01. Sample per second: 14753\n",
      "MF_BPR: Epoch 779 of 840. Elapsed time 54.70 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.59E+01. Sample per second: 14271\n",
      "MF_BPR: Epoch 780 of 840. Elapsed time 54.77 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.08 seconds. BPR loss 1.61E+01. Sample per second: 13768\n",
      "MF_BPR: Epoch 781 of 840. Elapsed time 54.84 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.25 seconds. BPR loss 1.60E+01. Sample per second: 16460\n",
      "MF_BPR: Epoch 782 of 840. Elapsed time 54.91 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 1.59E+01. Sample per second: 15876\n",
      "MF_BPR: Epoch 783 of 840. Elapsed time 54.98 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.60E+01. Sample per second: 15370\n",
      "MF_BPR: Epoch 784 of 840. Elapsed time 55.05 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 1.60E+01. Sample per second: 14935\n",
      "MF_BPR: Epoch 785 of 840. Elapsed time 55.11 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.82 seconds. BPR loss 1.60E+01. Sample per second: 14510\n",
      "MF_BPR: Epoch 786 of 840. Elapsed time 55.18 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.00 seconds. BPR loss 1.61E+01. Sample per second: 13974\n",
      "MF_BPR: Epoch 787 of 840. Elapsed time 55.25 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.17 seconds. BPR loss 1.60E+01. Sample per second: 16756\n",
      "MF_BPR: Epoch 788 of 840. Elapsed time 55.32 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.34 seconds. BPR loss 1.60E+01. Sample per second: 16099\n",
      "MF_BPR: Epoch 789 of 840. Elapsed time 55.39 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.50 seconds. BPR loss 1.60E+01. Sample per second: 15516\n",
      "MF_BPR: Epoch 790 of 840. Elapsed time 55.46 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 1.61E+01. Sample per second: 14911\n",
      "MF_BPR: Epoch 791 of 840. Elapsed time 55.53 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 1.61E+01. Sample per second: 14309\n",
      "MF_BPR: Epoch 792 of 840. Elapsed time 55.60 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.09 seconds. BPR loss 1.61E+01. Sample per second: 13722\n",
      "MF_BPR: Epoch 793 of 840. Elapsed time 55.67 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.30 seconds. BPR loss 1.61E+01. Sample per second: 16268\n",
      "MF_BPR: Epoch 794 of 840. Elapsed time 55.74 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.54 seconds. BPR loss 1.61E+01. Sample per second: 15404\n",
      "MF_BPR: Epoch 795 of 840. Elapsed time 55.81 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.62E+01. Sample per second: 14394\n",
      "MF_BPR: Epoch 796 of 840. Elapsed time 55.88 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.11 seconds. BPR loss 1.62E+01. Sample per second: 13687\n",
      "MF_BPR: Epoch 797 of 840. Elapsed time 55.95 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 1.61E+01. Sample per second: 15583\n",
      "MF_BPR: Epoch 798 of 840. Elapsed time 56.03 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.75 seconds. BPR loss 1.62E+01. Sample per second: 14709\n",
      "MF_BPR: Epoch 799 of 840. Elapsed time 56.10 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.96 seconds. BPR loss 1.62E+01. Sample per second: 14093\n",
      "MF_BPR: Epoch 800 of 840. Elapsed time 56.17 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.17 seconds. BPR loss 1.62E+01. Sample per second: 13523\n",
      "MF_BPR: Epoch 801 of 840. Elapsed time 56.24 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 1.62E+01. Sample per second: 15891\n",
      "MF_BPR: Epoch 802 of 840. Elapsed time 56.31 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.61 seconds. BPR loss 1.63E+01. Sample per second: 15144\n",
      "MF_BPR: Epoch 803 of 840. Elapsed time 56.38 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.81 seconds. BPR loss 1.62E+01. Sample per second: 14515\n",
      "MF_BPR: Epoch 804 of 840. Elapsed time 56.45 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.03 seconds. BPR loss 1.63E+01. Sample per second: 13899\n",
      "MF_BPR: Epoch 805 of 840. Elapsed time 56.52 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.22 seconds. BPR loss 1.63E+01. Sample per second: 16566\n",
      "MF_BPR: Epoch 806 of 840. Elapsed time 56.59 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.59 seconds. BPR loss 1.63E+01. Sample per second: 15208\n",
      "MF_BPR: Epoch 807 of 840. Elapsed time 56.66 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.95 seconds. BPR loss 1.63E+01. Sample per second: 14112\n",
      "MF_BPR: Epoch 808 of 840. Elapsed time 56.74 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.15 seconds. BPR loss 1.64E+01. Sample per second: 13558\n",
      "MF_BPR: Epoch 809 of 840. Elapsed time 56.81 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.46 seconds. BPR loss 1.63E+01. Sample per second: 15672\n",
      "MF_BPR: Epoch 810 of 840. Elapsed time 56.88 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.74 seconds. BPR loss 1.64E+01. Sample per second: 14756\n",
      "MF_BPR: Epoch 811 of 840. Elapsed time 56.95 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.10 seconds. BPR loss 1.64E+01. Sample per second: 13691\n",
      "MF_BPR: Epoch 812 of 840. Elapsed time 57.02 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.39 seconds. BPR loss 1.66E+01. Sample per second: 15914\n",
      "MF_BPR: Epoch 813 of 840. Elapsed time 57.09 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.58 seconds. BPR loss 1.63E+01. Sample per second: 15242\n",
      "MF_BPR: Epoch 814 of 840. Elapsed time 57.16 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.80 seconds. BPR loss 1.65E+01. Sample per second: 14556\n",
      "MF_BPR: Epoch 815 of 840. Elapsed time 57.23 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.04 seconds. BPR loss 1.64E+01. Sample per second: 13879\n",
      "MF_BPR: Epoch 816 of 840. Elapsed time 57.30 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 1.63E+01. Sample per second: 16286\n",
      "MF_BPR: Epoch 817 of 840. Elapsed time 57.37 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 1.65E+01. Sample per second: 15585\n",
      "MF_BPR: Epoch 818 of 840. Elapsed time 57.44 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.65E+01. Sample per second: 14866\n",
      "MF_BPR: Epoch 819 of 840. Elapsed time 57.51 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.88 seconds. BPR loss 1.66E+01. Sample per second: 14314\n",
      "MF_BPR: Epoch 820 of 840. Elapsed time 57.58 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.66E+01. Sample per second: 13831\n",
      "MF_BPR: Epoch 821 of 840. Elapsed time 57.65 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.23 seconds. BPR loss 1.66E+01. Sample per second: 16529\n",
      "MF_BPR: Epoch 822 of 840. Elapsed time 57.72 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.40 seconds. BPR loss 1.65E+01. Sample per second: 15900\n",
      "MF_BPR: Epoch 823 of 840. Elapsed time 57.79 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.55 seconds. BPR loss 1.65E+01. Sample per second: 15355\n",
      "MF_BPR: Epoch 824 of 840. Elapsed time 57.86 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.69 seconds. BPR loss 1.65E+01. Sample per second: 14904\n",
      "MF_BPR: Epoch 825 of 840. Elapsed time 57.93 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.85 seconds. BPR loss 1.66E+01. Sample per second: 14397\n",
      "MF_BPR: Epoch 826 of 840. Elapsed time 58.00 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.05 seconds. BPR loss 1.66E+01. Sample per second: 13843\n",
      "MF_BPR: Epoch 827 of 840. Elapsed time 58.07 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.28 seconds. BPR loss 1.67E+01. Sample per second: 16346\n",
      "MF_BPR: Epoch 828 of 840. Elapsed time 58.14 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.53 seconds. BPR loss 1.67E+01. Sample per second: 15415\n",
      "MF_BPR: Epoch 829 of 840. Elapsed time 58.21 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.66E+01. Sample per second: 14251\n",
      "MF_BPR: Epoch 830 of 840. Elapsed time 58.28 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.23 seconds. BPR loss 1.66E+01. Sample per second: 13353\n",
      "MF_BPR: Epoch 831 of 840. Elapsed time 58.36 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.49 seconds. BPR loss 1.66E+01. Sample per second: 15550\n",
      "MF_BPR: Epoch 832 of 840. Elapsed time 58.43 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.70 seconds. BPR loss 1.67E+01. Sample per second: 14859\n",
      "MF_BPR: Epoch 833 of 840. Elapsed time 58.50 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.90 seconds. BPR loss 1.67E+01. Sample per second: 14265\n",
      "MF_BPR: Epoch 834 of 840. Elapsed time 58.57 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.07 seconds. BPR loss 1.67E+01. Sample per second: 13776\n",
      "MF_BPR: Epoch 835 of 840. Elapsed time 58.64 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.29 seconds. BPR loss 1.67E+01. Sample per second: 16271\n",
      "MF_BPR: Epoch 836 of 840. Elapsed time 58.71 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.48 seconds. BPR loss 1.67E+01. Sample per second: 15612\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Epoch 837 of 840. Elapsed time 58.78 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.68 seconds. BPR loss 1.66E+01. Sample per second: 14935\n",
      "MF_BPR: Epoch 838 of 840. Elapsed time 58.85 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 4.91 seconds. BPR loss 1.69E+01. Sample per second: 14237\n",
      "MF_BPR: Epoch 839 of 840. Elapsed time 58.92 min\n",
      "MF_BPR: Processed 69878 ( 100.00% ) in 5.14 seconds. BPR loss 1.68E+01. Sample per second: 13601\n",
      "MF_BPR: Epoch 840 of 840. Elapsed time 58.99 min\n",
      "MF_BPR: Terminating at epoch 840. Elapsed time 58.99 min\n",
      "EvaluatorHoldout: Processed 52000 ( 74.51% ) in 30.06 sec. Users per second: 1730\n",
      "EvaluatorHoldout: Processed 69786 ( 100.00% ) in 39.86 sec. Users per second: 1751\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'sgd_mode': 'adam', 'epochs': 840, 'num_factors': 200, 'batch_size': 1, 'positive_reg': 2.1264549316860416e-05, 'negative_reg': 0.00033305173444924584, 'learning_rate': 0.00010151223842175566} - results:\n",
      "CUTOFF: 10 - ROC_AUC: 0.2780137, PRECISION: 0.1284226, PRECISION_RECALL_MIN_DEN: 0.1474181, RECALL: 0.0804195, MAP: 0.0873150, MRR: 0.2643487, NDCG: 0.0883011, F1: 0.0989042, HIT_RATE: 1.2842261, ARHR: 0.4349891, NOVELTY: 0.0099039, AVERAGE_POPULARITY: 0.4711777, DIVERSITY_MEAN_INTER_LIST: 0.9225004, DIVERSITY_HERFINDAHL: 0.9922487, COVERAGE_ITEM: 0.9872671, COVERAGE_ITEM_CORRECT: 0.0668477, COVERAGE_USER: 0.9986834, COVERAGE_USER_CORRECT: 0.4733965, DIVERSITY_GINI: 0.1365917, SHANNON_ENTROPY: 9.0424833, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/MatrixFactorization_BPR_Cython_Recommender\n",
      "\n",
      "MatrixFactorization_BPR_Cython_Recommender: Saving model in file 'result_experiments/MatrixFactorization_BPR_Cython_Recommender_best_model_last'\n",
      "MatrixFactorization_BPR_Cython_Recommender: Saving complete\n"
     ]
    }
   ],
   "source": [
    "parameterSearch.search(recommender_input_args,\n",
    "                       recommender_input_args_last_test = recommender_input_args_last_test,\n",
    "                       parameter_search_space = hyperparameters_range_dictionary,\n",
    "                       n_cases = n_cases,\n",
    "                       n_random_starts = n_random_starts,\n",
    "                       save_model = \"last\",\n",
    "                       output_folder_path = output_folder_path,\n",
    "                       output_file_name_root = recommender_class.RECOMMENDER_NAME,\n",
    "                       metric_to_optimize = metric_to_optimize\n",
    "                      )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'sgd_mode': 'adam',\n",
       "  'epochs': 1500,\n",
       "  'num_factors': 9,\n",
       "  'batch_size': 256,\n",
       "  'positive_reg': 0.0009557358739689882,\n",
       "  'negative_reg': 0.0013254069621555342,\n",
       "  'learning_rate': 0.025468411698118695},\n",
       " {'sgd_mode': 'adagrad',\n",
       "  'epochs': 1500,\n",
       "  'num_factors': 200,\n",
       "  'batch_size': 64,\n",
       "  'positive_reg': 1e-05,\n",
       "  'negative_reg': 1e-05,\n",
       "  'learning_rate': 0.0001},\n",
       " {'sgd_mode': 'sgd',\n",
       "  'epochs': 1500,\n",
       "  'num_factors': 1,\n",
       "  'batch_size': 16,\n",
       "  'positive_reg': 0.01,\n",
       "  'negative_reg': 0.01,\n",
       "  'learning_rate': 0.1},\n",
       " {'sgd_mode': 'adam',\n",
       "  'epochs': 1500,\n",
       "  'num_factors': 199,\n",
       "  'batch_size': 256,\n",
       "  'positive_reg': 1e-05,\n",
       "  'negative_reg': 1e-05,\n",
       "  'learning_rate': 0.0001},\n",
       " {'sgd_mode': 'adam',\n",
       "  'epochs': 1500,\n",
       "  'num_factors': 200,\n",
       "  'batch_size': 1,\n",
       "  'positive_reg': 2.1264549316860416e-05,\n",
       "  'negative_reg': 0.00033305173444924584,\n",
       "  'learning_rate': 0.00010151223842175566}]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_metadata = data_loader.load_data(recommender_class.RECOMMENDER_NAME + \"_metadata.zip\")\n",
    "\n",
    "hyperparameters_list = search_metadata[\"hyperparameters_list\"]\n",
    "hyperparameters_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'sgd_mode': 'adam',\n",
       " 'epochs': 840,\n",
       " 'num_factors': 200,\n",
       " 'batch_size': 1,\n",
       " 'positive_reg': 2.1264549316860416e-05,\n",
       " 'negative_reg': 0.00033305173444924584,\n",
       " 'learning_rate': 0.00010151223842175566}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_parameters = search_metadata[\"hyperparameters_best\"]\n",
    "best_parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to use the predefined hyperparameter range\n",
    "\n",
    "#### Function runParameterSearch_Collaborative takes as input all needed to optimize any of the recommenders in the framework as well as other parameters such as which similarity heuristics to use for the KNNs, whether to parallelize the training of KNNs and so on..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity_type_list = ['cosine', 'jaccard', \"asymmetric\", \"dice\", \"tversky\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SearchSingleCase: Resuming 'TopPopRecommender' Failed, no such file exists.\n",
      "TopPopRecommender: URM Detected 73 (0.68 %) cold items.\n",
      "SearchSingleCase: Testing config: {}\n",
      "EvaluatorHoldout: Processed 66000 ( 94.78% ) in 30.25 sec. Users per second: 2182\n",
      "EvaluatorHoldout: Processed 69635 ( 100.00% ) in 31.89 sec. Users per second: 2183\n",
      "SearchSingleCase: New best config found. Config 0: {} - results: ROC_AUC: 0.3147372, PRECISION: 0.1120011, PRECISION_RECALL_MIN_DEN: 0.1336714, RECALL: 0.0782386, MAP: 0.0636972, MRR: 0.2674676, NDCG: 0.0895075, F1: 0.0921239, HIT_RATE: 1.1200115, ARHR: 0.3775154, NOVELTY: 0.0078829, AVERAGE_POPULARITY: 0.8419826, DIVERSITY_MEAN_INTER_LIST: 0.3943088, DIVERSITY_HERFINDAHL: 0.9394303, COVERAGE_ITEM: 0.0044940, COVERAGE_ITEM_CORRECT: 0.0042131, COVERAGE_USER: 0.9965225, COVERAGE_USER_CORRECT: 0.5519047, DIVERSITY_GINI: 0.0015082, SHANNON_ENTROPY: 4.2380186, \n",
      "\n",
      "SearchSingleCase: Saving model in result_experiments/TopPopRecommender\n",
      "\n",
      "TopPopRecommender: Saving model in file 'result_experiments/TopPopRecommender_best_model'\n",
      "TopPopRecommender: Saving complete\n",
      "EvaluatorHoldout: Processed 66000 ( 94.57% ) in 30.18 sec. Users per second: 2187\n",
      "EvaluatorHoldout: Processed 69786 ( 100.00% ) in 31.88 sec. Users per second: 2189\n",
      "SearchSingleCase: Best config evaluated with evaluator_test. Config: {} - results:\n",
      "CUTOFF: 10 - ROC_AUC: 0.3463799, PRECISION: 0.1401413, PRECISION_RECALL_MIN_DEN: 0.1559810, RECALL: 0.0781510, MAP: 0.0813803, MRR: 0.3154301, NDCG: 0.0978337, F1: 0.1003442, HIT_RATE: 1.4014129, ARHR: 0.4754139, NOVELTY: 0.0078828, AVERAGE_POPULARITY: 0.8420412, DIVERSITY_MEAN_INTER_LIST: 0.3940077, DIVERSITY_HERFINDAHL: 0.9394002, COVERAGE_ITEM: 0.0044940, COVERAGE_ITEM_CORRECT: 0.0044940, COVERAGE_USER: 0.9986834, COVERAGE_USER_CORRECT: 0.6037379, DIVERSITY_GINI: 0.0015076, SHANNON_ENTROPY: 4.2373597, \n",
      "\n",
      "\n",
      "DataIO: Json dumps supports only 'str' as dictionary keys. Transforming keys to string, note that this will alter the mapper content.\n",
      "SearchSingleCase: Search complete. Best config is 0: {}\n",
      "\n",
      "TopPopRecommender: URM Detected 37 (0.35 %) cold items.\n",
      "SearchSingleCase: Evaluation with constructor data for final test. Using best config: {}\n",
      "EvaluatorHoldout: Processed 65000 ( 93.14% ) in 30.03 sec. Users per second: 2165\n",
      "EvaluatorHoldout: Processed 69786 ( 100.00% ) in 32.39 sec. Users per second: 2154\n",
      "SearchSingleCase: Best config evaluated with evaluator_test with constructor data for final test. Config: {} - results:\n",
      "CUTOFF: 10 - ROC_AUC: 0.3754048, PRECISION: 0.1670220, PRECISION_RECALL_MIN_DEN: 0.1839800, RECALL: 0.0865500, MAP: 0.1122267, MRR: 0.3653918, NDCG: 0.1128662, F1: 0.1140170, HIT_RATE: 1.6702204, ARHR: 0.5925419, NOVELTY: 0.0079141, AVERAGE_POPULARITY: 0.8236519, DIVERSITY_MEAN_INTER_LIST: 0.4809575, DIVERSITY_HERFINDAHL: 0.9480951, COVERAGE_ITEM: 0.0079581, COVERAGE_ITEM_CORRECT: 0.0079581, COVERAGE_USER: 0.9986834, COVERAGE_USER_CORRECT: 0.6183777, DIVERSITY_GINI: 0.0017645, SHANNON_ENTROPY: 4.5329173, \n",
      "\n",
      "\n",
      "SearchSingleCase: Saving model in result_experiments/TopPopRecommender\n",
      "\n",
      "TopPopRecommender: Saving model in file 'result_experiments/TopPopRecommender_best_model_last'\n",
      "TopPopRecommender: Saving complete\n"
     ]
    }
   ],
   "source": [
    "from ParameterTuning.run_parameter_search import runParameterSearch_Collaborative, runParameterSearch_Content\n",
    "from Base.NonPersonalizedRecommender import TopPop, Random\n",
    "from GraphBased.P3alphaRecommender import P3alphaRecommender\n",
    "\n",
    "recommender_class = TopPop\n",
    "\n",
    "runParameterSearch_Collaborative(recommender_class,\n",
    "       URM_train = URM_train,\n",
    "       URM_train_last_test = URM_train_validation,\n",
    "       metric_to_optimize = metric_to_optimize,\n",
    "       evaluator_validation_earlystopping = evaluator_validation,\n",
    "       evaluator_validation = evaluator_validation,\n",
    "       evaluator_test = evaluator_test,\n",
    "       output_folder_path = output_folder_path,\n",
    "       parallelizeKNN = True,\n",
    "       allow_weighting = True,\n",
    "       resume_from_saved = True,\n",
    "       save_model = \"best\",\n",
    "       similarity_type_list = ['cosine', 'jaccard', \"asymmetric\", \"dice\", \"tversky\"],\n",
    "       n_cases = n_cases,\n",
    "       n_random_starts = n_random_starts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The process is similar for content-based algorithms.\n",
    "* Add the ICM data and its name (it will be used for file names)\n",
    "* Remove the evaluator for earlystopping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from KNN.ItemKNNCBFRecommender import ItemKNNCBFRecommender\n",
    "\n",
    "recommender_class = ItemKNNCBFRecommender\n",
    "\n",
    "runParameterSearch_Content(recommender_class,\n",
    "       URM_train = URM_train,\n",
    "       ICM_object = ICM_all,\n",
    "       ICM_name = \"ICM_genres\",\n",
    "       URM_train_last_test = URM_train_validation,\n",
    "       metric_to_optimize = metric_to_optimize,\n",
    "       evaluator_validation = evaluator_validation,\n",
    "       evaluator_test = evaluator_test,\n",
    "       output_folder_path = output_folder_path,\n",
    "       parallelizeKNN = True,\n",
    "       allow_weighting = True,\n",
    "       resume_from_saved = True,\n",
    "       save_model = \"best\",\n",
    "       similarity_type_list = ['cosine', 'jaccard', \"dice\", \"tversky\"],\n",
    "       n_cases = n_cases,\n",
    "       n_random_starts = n_random_starts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You can use the \"partial\" function to pre-set all attributes that are needed and use it to loop on the recommender_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Base.NonPersonalizedRecommender import TopPop, Random\n",
    "from GraphBased.P3alphaRecommender import P3alphaRecommender\n",
    "import os, multiprocessing\n",
    "from functools import partial\n",
    "\n",
    "\n",
    "runParameterSearch_Collaborative_partial = partial(runParameterSearch_Collaborative,\n",
    "       URM_train = URM_train,\n",
    "       URM_train_last_test = URM_train_validation,\n",
    "       metric_to_optimize = metric_to_optimize,\n",
    "       evaluator_validation_earlystopping = evaluator_validation,\n",
    "       evaluator_validation = evaluator_validation,\n",
    "       evaluator_test = evaluator_test,\n",
    "       output_folder_path = output_folder_path,\n",
    "       parallelizeKNN = True,\n",
    "       allow_weighting = True,\n",
    "       resume_from_saved = True,\n",
    "       save_model = \"best\",\n",
    "       similarity_type_list = ['cosine', 'jaccard', \"asymmetric\", \"dice\", \"tversky\"],\n",
    "       n_cases = n_cases,\n",
    "       n_random_starts = n_random_starts)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "collaborative_algorithm_list = [\n",
    "    Random,\n",
    "    TopPop,\n",
    "    P3alphaRecommender,\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You can call it as part of a parallel pool, hence parallizing the optimization of the various models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None, None, None]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pool = multiprocessing.Pool(processes=int(multiprocessing.cpu_count()), maxtasksperchild=1)\n",
    "pool.map(runParameterSearch_Collaborative_partial, collaborative_algorithm_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Or sequentially, note that since resume_from_saved = True this new call to the function will just load the optimized values and not re-run it "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SearchSingleCase: Resuming 'RandomRecommender'... Loaded 1 configurations.\n",
      "RandomRecommender: URM Detected 37 (0.35 %) cold items.\n",
      "SearchSingleCase: Resuming 'RandomRecommender'... Result on last already available.\n",
      "SearchSingleCase: Resuming 'TopPopRecommender'... Loaded 1 configurations.\n",
      "TopPopRecommender: URM Detected 37 (0.35 %) cold items.\n",
      "SearchSingleCase: Resuming 'TopPopRecommender'... Result on last already available.\n",
      "SearchBayesianSkopt: Resuming 'P3alphaRecommender'... Loaded 5 configurations.\n",
      "Iteration No: 1 started. Searching for the next optimal point.\n",
      "Iteration No: 1 ended. Search finished for the next optimal point.\n",
      "Time taken: 0.2288\n",
      "Function value obtained: -0.1047\n",
      "Current minimum: -0.1129\n",
      "P3alphaRecommender: URM Detected 37 (0.35 %) cold items.\n",
      "SearchBayesianSkopt: Resuming 'P3alphaRecommender'... Result on last already available.\n"
     ]
    }
   ],
   "source": [
    "for recommender_class in collaborative_algorithm_list:\n",
    "\n",
    "    try:\n",
    "\n",
    "        runParameterSearch_Collaborative_partial(recommender_class)\n",
    "\n",
    "    except Exception as e:\n",
    "\n",
    "        print(\"On recommender {} Exception {}\".format(recommender_class, str(e)))\n",
    "        traceback.print_exc()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to load and save a model\n",
    "### The models are saved in two files:\n",
    "* RECOMMENDER_NAME + \"_best_model.zip\": contains the model trained on the URM_train\n",
    "* RECOMMENDER_NAME + \"_best_model_last.zip\": contains the model trained on the URM_train_validation\n",
    "\n",
    "#### You can load them using the \"load_model\" function of the Recommender class, to do so:\n",
    "* Create an instance of the recommender you want to load and provide the needed URM data and ICM data, if needed. The URM you pass is the one that will be used as user profiles to generate the recommendations.\n",
    "* Given the instance, call the load_model function\n",
    "* You can now use the recommender to generate recommendations.\n",
    "\n",
    "#### If you want to save your own models use the \"save_model\" function, it takes thesame parameters as \"load_model\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P3alphaRecommender: URM Detected 4 (0.04 %) cold items.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<GraphBased.P3alphaRecommender.P3alphaRecommender at 0x14a15dca780>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from GraphBased.P3alphaRecommender import P3alphaRecommender\n",
    "\n",
    "recommender_object = P3alphaRecommender(URM_all)\n",
    "recommender_object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P3alphaRecommender: Loading model from file 'result_experiments/P3alphaRecommender_best_model_last.zip'\n",
      "P3alphaRecommender: Loading complete\n"
     ]
    }
   ],
   "source": [
    "recommender_object.load_model(output_folder_path, \n",
    "                              file_name = recommender_object.RECOMMENDER_NAME + \"_best_model_last.zip\" )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<10681x10681 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 2666401 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommender_object.W_sparse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[19, 22, 7, 175, 178, 139, 605, 404, 133, 1293]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_id = 10\n",
    "recommender_object.recommend(user_id, cutoff = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P3alphaRecommender: Saving model in file 'result_experiments/P3alphaRecommender_my_own_save.zip'\n",
      "P3alphaRecommender: Saving complete\n"
     ]
    }
   ],
   "source": [
    "recommender_object.save_model(output_folder_path, \n",
    "                              file_name = recommender_object.RECOMMENDER_NAME + \"_my_own_save.zip\" )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
